[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:17:45 dist-test-slave-g6r4 krb5kdc[400](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:17:45 dist-test-slave-g6r4 krb5kdc[400](info): set up 2 sockets
Jun 18 22:17:45 dist-test-slave-g6r4 krb5kdc[400](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260618 22:17:46.470440 393 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/krb5kdc/test-user.keytab.
Jun 18 22:17:46 dist-test-slave-g6r4 krb5kdc[400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:17:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:17:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260618 22:17:46.589067 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:40895
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:41139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:40895
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:17:46.704421 413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:17:46.704684 413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:17:46.704725 413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:17:46.708721 413 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:17:46.708772 413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:17:46.708791 413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:17:46.708808 413 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:17:46.708823 413 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:17:46.713862 413 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:41139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:40895
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:40895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:17:46.714886 413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:17:46.715862 413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:17:46.724992 419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:17:46.725025 418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:17:46.725041 421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:17:46.725086 413 server_base.cc:1034] running on GCE node
I20260618 22:17:46.725709 413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:17:46.726804 413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:17:46.728015 413 hybrid_clock.cc:648] HybridClock initialized: now 1781821066727941 us; error 88 us; skew 500 ppm
Jun 18 22:17:46 dist-test-slave-g6r4 krb5kdc[400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821066, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:17:46.730712 413 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260618 22:17:46.731601 413 webserver.cc:460] Webserver started at http://127.0.98.126:46013/ using document root <none> and password file <none>
I20260618 22:17:46.732156 413 fs_manager.cc:359] Metadata directory not provided
I20260618 22:17:46.732254 413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:17:46.732491 413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:17:46.734548 413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/minicluster-data/master-0/data/instance:
uuid: "1655aa507dc741cfbcd4f124da12c5b1"
format_stamp: "Formatted at 2026-06-18 22:17:46 on dist-test-slave-g6r4"
server_key: "cde0096eca8ff6db5256116590574eb1"
server_key_iv: "551152066782ffd6d32cb9328adefa2d"
server_key_version: "encryptionkey@0"
I20260618 22:17:46.735122 413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/minicluster-data/master-0/wal/instance:
uuid: "1655aa507dc741cfbcd4f124da12c5b1"
format_stamp: "Formatted at 2026-06-18 22:17:46 on dist-test-slave-g6r4"
server_key: "cde0096eca8ff6db5256116590574eb1"
server_key_iv: "551152066782ffd6d32cb9328adefa2d"
server_key_version: "encryptionkey@0"
I20260618 22:17:46.738730 413 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:17:46.741074 428 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:17:46.742432 413 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:17:46.742596 413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/minicluster-data/master-0/wal
uuid: "1655aa507dc741cfbcd4f124da12c5b1"
format_stamp: "Formatted at 2026-06-18 22:17:46 on dist-test-slave-g6r4"
server_key: "cde0096eca8ff6db5256116590574eb1"
server_key_iv: "551152066782ffd6d32cb9328adefa2d"
server_key_version: "encryptionkey@0"
I20260618 22:17:46.742733 413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:17:46.758586 413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:17:46.761416 413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:17:46.761636 413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:17:46.768323 413 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:40895
I20260618 22:17:46.768339 480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:40895 every 8 connection(s)
I20260618 22:17:46.769268 413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
I20260618 22:17:46.772011 481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:17:46.776094 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 413
I20260618 22:17:46.776266 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/minicluster-data/master-0/wal/instance
I20260618 22:17:46.777843 393 external_mini_cluster.cc:1428] Setting key e7ca2344e0a5dcf1787c3b4fba7d649b
I20260618 22:17:46.777887 481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1655aa507dc741cfbcd4f124da12c5b1: Bootstrap starting.
I20260618 22:17:46.780764 481 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1655aa507dc741cfbcd4f124da12c5b1: Neither blocks nor log segments found. Creating new log.
I20260618 22:17:46.781677 481 log.cc:824] T 00000000000000000000000000000000 P 1655aa507dc741cfbcd4f124da12c5b1: Log is configured to *not* fsync() on all Append() calls
I20260618 22:17:46.783618 481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1655aa507dc741cfbcd4f124da12c5b1: No bootstrap required, opened a new log
I20260618 22:17:46.787691 481 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1655aa507dc741cfbcd4f124da12c5b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1655aa507dc741cfbcd4f124da12c5b1" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40895 } }
I20260618 22:17:46.787971 481 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1655aa507dc741cfbcd4f124da12c5b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:17:46.788030 481 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1655aa507dc741cfbcd4f124da12c5b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1655aa507dc741cfbcd4f124da12c5b1, State: Initialized, Role: FOLLOWER
I20260618 22:17:46.788483 481 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1655aa507dc741cfbcd4f124da12c5b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1655aa507dc741cfbcd4f124da12c5b1" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40895 } }
I20260618 22:17:46.788616 481 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1655aa507dc741cfbcd4f124da12c5b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:17:46.788707 481 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1655aa507dc741cfbcd4f124da12c5b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:17:46.788841 481 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1655aa507dc741cfbcd4f124da12c5b1 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:17:46.789728 481 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1655aa507dc741cfbcd4f124da12c5b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1655aa507dc741cfbcd4f124da12c5b1" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40895 } }
I20260618 22:17:46.790083 481 leader_election.cc:302] T 00000000000000000000000000000000 P 1655aa507dc741cfbcd4f124da12c5b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1655aa507dc741cfbcd4f124da12c5b1; no voters:
I20260618 22:17:46.790410 481 leader_election.cc:288] T 00000000000000000000000000000000 P 1655aa507dc741cfbcd4f124da12c5b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:17:46.790536 486 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1655aa507dc741cfbcd4f124da12c5b1 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:17:46.790791 486 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1655aa507dc741cfbcd4f124da12c5b1 [term 1 LEADER]: Becoming Leader. State: Replica: 1655aa507dc741cfbcd4f124da12c5b1, State: Running, Role: LEADER
I20260618 22:17:46.791064 486 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1655aa507dc741cfbcd4f124da12c5b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1655aa507dc741cfbcd4f124da12c5b1" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40895 } }
I20260618 22:17:46.791673 481 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1655aa507dc741cfbcd4f124da12c5b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:17:46.792672 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1655aa507dc741cfbcd4f124da12c5b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1655aa507dc741cfbcd4f124da12c5b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1655aa507dc741cfbcd4f124da12c5b1" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40895 } } }
I20260618 22:17:46.792670 488 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1655aa507dc741cfbcd4f124da12c5b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1655aa507dc741cfbcd4f124da12c5b1. Latest consensus state: current_term: 1 leader_uuid: "1655aa507dc741cfbcd4f124da12c5b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1655aa507dc741cfbcd4f124da12c5b1" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40895 } } }
I20260618 22:17:46.792842 488 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1655aa507dc741cfbcd4f124da12c5b1 [sys.catalog]: This master's current role is: LEADER
I20260618 22:17:46.792842 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1655aa507dc741cfbcd4f124da12c5b1 [sys.catalog]: This master's current role is: LEADER
I20260618 22:17:46.793521 495 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:17:46.797118 495 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 18 22:17:46 dist-test-slave-g6r4 krb5kdc[400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:17:46.803431 495 catalog_manager.cc:1347] Generated new cluster ID: ddb2f9c01aef40eca623e803448f4f18
I20260618 22:17:46.803498 495 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:17:46.804598 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:46.781275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34622 (local address 127.0.98.126:40895)
0618 22:17:46.781771 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0618 22:17:46.781792 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0618 22:17:46.783073 (+ 1281us) server_negotiation.cc:407] Connection header received
0618 22:17:46.787227 (+ 4154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:17:46.787249 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:17:46.787625 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:17:46.787976 (+ 351us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:17:46.790619 (+ 2643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:46.792084 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:46.793967 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:46.794683 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:46.794861 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:46.798729 (+ 3868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:17:46.798734 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:17:46.798747 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:17:46.798773 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:17:46.800920 (+ 2147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:17:46.801587 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:17:46.801593 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:17:46.801598 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:17:46.801733 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:17:46.802169 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:17:46.802173 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:17:46.802175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:17:46.802695 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:17:46.802920 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0618 22:17:46.803723 (+ 803us) server_negotiation.cc:299] Negotiation successful
0618 22:17:46.803951 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":132,"threads_started":1}
I20260618 22:17:46.831154 495 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:17:46.832206 495 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:17:46.838667 495 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1655aa507dc741cfbcd4f124da12c5b1: Generated new TSK 0
I20260618 22:17:46.839462 495 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab.
I20260618 22:17:46.874032 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:40895
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:41139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:17:46.991194 507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:17:46.991423 507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:17:46.991463 507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:17:46.995590 507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:17:46.995642 507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:17:46.995735 507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260618 22:17:47.000864 507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:41139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:40895
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:17:47.001868 507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:17:47.002903 507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:17:47.011909 512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:17:47.012045 507 server_base.cc:1034] running on GCE node
W20260618 22:17:47.011915 515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:17:47.011927 513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:17:47.012588 507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:17:47.013214 507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:17:47.014459 507 hybrid_clock.cc:648] HybridClock initialized: now 1781821067014417 us; error 60 us; skew 500 ppm
Jun 18 22:17:47 dist-test-slave-g6r4 krb5kdc[400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821067, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:17:47.017201 507 init.cc:377] Logged in from keytab as kudu/127.0.98.65@KRBTEST.COM (short username kudu)
I20260618 22:17:47.018115 507 webserver.cc:460] Webserver started at http://127.0.98.65:35095/ using document root <none> and password file <none>
I20260618 22:17:47.018642 507 fs_manager.cc:359] Metadata directory not provided
I20260618 22:17:47.018743 507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:17:47.018965 507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:17:47.020928 507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/minicluster-data/ts-0/data/instance:
uuid: "39b164da52d949dcb783045478492fab"
format_stamp: "Formatted at 2026-06-18 22:17:47 on dist-test-slave-g6r4"
server_key: "195416440ddb8ef160cc455ddbe97680"
server_key_iv: "626f87c54128be27f2fb5747826201b0"
server_key_version: "encryptionkey@0"
I20260618 22:17:47.021502 507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/minicluster-data/ts-0/wal/instance:
uuid: "39b164da52d949dcb783045478492fab"
format_stamp: "Formatted at 2026-06-18 22:17:47 on dist-test-slave-g6r4"
server_key: "195416440ddb8ef160cc455ddbe97680"
server_key_iv: "626f87c54128be27f2fb5747826201b0"
server_key_version: "encryptionkey@0"
I20260618 22:17:47.025018 507 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260618 22:17:47.027393 522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:17:47.028528 507 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260618 22:17:47.028676 507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/minicluster-data/ts-0/wal
uuid: "39b164da52d949dcb783045478492fab"
format_stamp: "Formatted at 2026-06-18 22:17:47 on dist-test-slave-g6r4"
server_key: "195416440ddb8ef160cc455ddbe97680"
server_key_iv: "626f87c54128be27f2fb5747826201b0"
server_key_version: "encryptionkey@0"
I20260618 22:17:47.028785 507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:17:47.049312 507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:17:47.052546 507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:17:47.052757 507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:17:47.053360 507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:17:47.054204 507 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:17:47.054281 507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:17:47.054348 507 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:17:47.054386 507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:17:47.063130 507 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:44977
I20260618 22:17:47.063155 635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:44977 every 8 connection(s)
I20260618 22:17:47.063998 507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
I20260618 22:17:47.070433 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 507
I20260618 22:17:47.070555 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/minicluster-data/ts-0/wal/instance
I20260618 22:17:47.070842 393 external_mini_cluster.cc:1428] Setting key 337e3c6e27f1a4db4ae66f77f1c35caa
Jun 18 22:17:47 dist-test-slave-g6r4 krb5kdc[400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821067, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:17:47.078373 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:47.066014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:58147 (local address 127.0.98.126:40895)
0618 22:17:47.066136 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0618 22:17:47.066139 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:17:47.066955 (+ 816us) server_negotiation.cc:407] Connection header received
0618 22:17:47.068307 (+ 1352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:17:47.068310 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:17:47.068367 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:17:47.068479 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:17:47.069604 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:47.070219 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:47.071633 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:47.071900 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:47.072021 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:47.074646 (+ 2625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:17:47.074649 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:17:47.074651 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:17:47.074673 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:17:47.076130 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:17:47.076766 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:17:47.076769 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:17:47.076770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:17:47.076841 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:17:47.077256 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:17:47.077258 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:17:47.077259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:17:47.077418 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:17:47.077491 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0618 22:17:47.078149 (+ 658us) server_negotiation.cc:299] Negotiation successful
0618 22:17:47.078242 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":32}
I20260618 22:17:47.079536 638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:47.066377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:40895 (local address 127.0.98.65:58147)
0618 22:17:47.066803 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0618 22:17:47.066844 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0618 22:17:47.068075 (+ 1231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:17:47.068647 (+ 572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:17:47.068654 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:17:47.069002 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:17:47.069451 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:47.069464 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:47.070347 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:47.070350 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:47.071495 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:47.071504 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:47.072054 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:47.072058 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:47.072229 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:47.072828 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:17:47.072843 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:17:47.074431 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:17:47.076279 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:17:47.076288 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:17:47.076305 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:17:47.076665 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:17:47.076949 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:17:47.076955 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:17:47.076960 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:17:47.077152 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:17:47.077529 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:17:47.077539 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:17:47.077868 (+ 329us) client_negotiation.cc:769] Sending connection context
0618 22:17:47.078125 (+ 257us) client_negotiation.cc:240] Negotiation successful
0618 22:17:47.078478 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":99,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.66@KRBTEST.COM; defaulting to no policy
I20260618 22:17:47.080817 636 heartbeater.cc:347] Connected to a master server at 127.0.98.126:40895
I20260618 22:17:47.081049 636 heartbeater.cc:464] Registering TS with master...
I20260618 22:17:47.081465 636 heartbeater.cc:511] Master 127.0.98.126:40895 requested a full tablet report, sending...
I20260618 22:17:47.082978 445 ts_manager.cc:194] Registered new tserver with Master: 39b164da52d949dcb783045478492fab (127.0.98.65:44977)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.66@KRBTEST.COM" created.
I20260618 22:17:47.084501 445 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.65@KRBTEST.COM'} at 127.0.98.65:58147
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab.
I20260618 22:17:47.099293 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:40895
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:41139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:17:47.207435 641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:17:47.207667 641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:17:47.207708 641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:17:47.211949 641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:17:47.212003 641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:17:47.212100 641 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260618 22:17:47.217156 641 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:41139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:40895
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:17:47.218151 641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:17:47.219105 641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:17:47.228335 646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:17:47.228335 647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:17:47.228356 649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:17:47.228674 641 server_base.cc:1034] running on GCE node
I20260618 22:17:47.229070 641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:17:47.229640 641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:17:47.230831 641 hybrid_clock.cc:648] HybridClock initialized: now 1781821067230823 us; error 28 us; skew 500 ppm
Jun 18 22:17:47 dist-test-slave-g6r4 krb5kdc[400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821067, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:17:47.233646 641 init.cc:377] Logged in from keytab as kudu/127.0.98.66@KRBTEST.COM (short username kudu)
I20260618 22:17:47.234526 641 webserver.cc:460] Webserver started at http://127.0.98.66:32791/ using document root <none> and password file <none>
I20260618 22:17:47.235023 641 fs_manager.cc:359] Metadata directory not provided
I20260618 22:17:47.235078 641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:17:47.235230 641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:17:47.237111 641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/minicluster-data/ts-1/data/instance:
uuid: "09988dd58aa44e2188a6eb67d52ec6a3"
format_stamp: "Formatted at 2026-06-18 22:17:47 on dist-test-slave-g6r4"
server_key: "46064b442b90edb4112ba2ab26086cc9"
server_key_iv: "04ed689456ba70a97ae7cccb24c3fee6"
server_key_version: "encryptionkey@0"
I20260618 22:17:47.237607 641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/minicluster-data/ts-1/wal/instance:
uuid: "09988dd58aa44e2188a6eb67d52ec6a3"
format_stamp: "Formatted at 2026-06-18 22:17:47 on dist-test-slave-g6r4"
server_key: "46064b442b90edb4112ba2ab26086cc9"
server_key_iv: "04ed689456ba70a97ae7cccb24c3fee6"
server_key_version: "encryptionkey@0"
I20260618 22:17:47.241276 641 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260618 22:17:47.243831 656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:17:47.245007 641 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:17:47.245151 641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/minicluster-data/ts-1/wal
uuid: "09988dd58aa44e2188a6eb67d52ec6a3"
format_stamp: "Formatted at 2026-06-18 22:17:47 on dist-test-slave-g6r4"
server_key: "46064b442b90edb4112ba2ab26086cc9"
server_key_iv: "04ed689456ba70a97ae7cccb24c3fee6"
server_key_version: "encryptionkey@0"
I20260618 22:17:47.245260 641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:17:47.266557 641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:17:47.269240 641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:17:47.269434 641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:17:47.269971 641 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:17:47.270733 641 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:17:47.270809 641 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:17:47.270870 641 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:17:47.270907 641 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:17:47.280265 641 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:40587
I20260618 22:17:47.280339 769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:40587 every 8 connection(s)
I20260618 22:17:47.281148 641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
I20260618 22:17:47.285017 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 641
I20260618 22:17:47.285154 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/minicluster-data/ts-1/wal/instance
I20260618 22:17:47.285434 393 external_mini_cluster.cc:1428] Setting key 6c2c616e01bac79e3b0188810c2246e3
Jun 18 22:17:47 dist-test-slave-g6r4 krb5kdc[400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821067, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.67@KRBTEST.COM; defaulting to no policy
I20260618 22:17:47.295243 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:47.282946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:59457 (local address 127.0.98.126:40895)
0618 22:17:47.283092 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0618 22:17:47.283097 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:17:47.283962 (+ 865us) server_negotiation.cc:407] Connection header received
0618 22:17:47.285287 (+ 1325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:17:47.285292 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:17:47.285347 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:17:47.285437 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:17:47.286699 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:47.287259 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:47.288539 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:47.288771 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:47.288889 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:47.291454 (+ 2565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:17:47.291458 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:17:47.291461 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:17:47.291489 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:17:47.293122 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:17:47.293646 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:17:47.293651 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:17:47.293653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:17:47.293750 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:17:47.294135 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:17:47.294140 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:17:47.294142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:17:47.294314 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:17:47.294424 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0618 22:17:47.294963 (+ 539us) server_negotiation.cc:299] Negotiation successful
0618 22:17:47.295077 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260618 22:17:47.296000 772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:47.283360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:40895 (local address 127.0.98.66:59457)
0618 22:17:47.283810 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0618 22:17:47.283851 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0618 22:17:47.285025 (+ 1174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:17:47.285669 (+ 644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:17:47.285676 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:17:47.286075 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:17:47.286525 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:47.286557 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:47.287403 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:47.287407 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:47.288274 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:47.288283 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:47.288932 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:47.288937 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:47.289107 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:47.289739 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:17:47.289754 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:17:47.291287 (+ 1533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:17:47.293263 (+ 1976us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:17:47.293269 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:17:47.293282 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:17:47.293536 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:17:47.293854 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:17:47.293857 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:17:47.293859 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:17:47.294026 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:17:47.294433 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:17:47.294439 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:17:47.294730 (+ 291us) client_negotiation.cc:769] Sending connection context
0618 22:17:47.294911 (+ 181us) client_negotiation.cc:240] Negotiation successful
0618 22:17:47.295210 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":99,"threads_started":1}
I20260618 22:17:47.297221 770 heartbeater.cc:347] Connected to a master server at 127.0.98.126:40895
I20260618 22:17:47.297451 770 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.67@KRBTEST.COM" created.
I20260618 22:17:47.298023 770 heartbeater.cc:511] Master 127.0.98.126:40895 requested a full tablet report, sending...
I20260618 22:17:47.299082 445 ts_manager.cc:194] Registered new tserver with Master: 09988dd58aa44e2188a6eb67d52ec6a3 (127.0.98.66:40587)
I20260618 22:17:47.299642 445 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.66@KRBTEST.COM'} at 127.0.98.66:59457
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab.
I20260618 22:17:47.312674 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:40895
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:41139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:17:47.419294 775 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:17:47.419544 775 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:17:47.419586 775 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:17:47.423666 775 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:17:47.423731 775 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:17:47.423840 775 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260618 22:17:47.429031 775 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:41139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:40895
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.775
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:17:47.430030 775 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:17:47.430994 775 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:17:47.440611 780 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:17:47.440675 783 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:17:47.440737 781 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:17:47.440845 775 server_base.cc:1034] running on GCE node
I20260618 22:17:47.441365 775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:17:47.442080 775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:17:47.443269 775 hybrid_clock.cc:648] HybridClock initialized: now 1781821067443253 us; error 28 us; skew 500 ppm
Jun 18 22:17:47 dist-test-slave-g6r4 krb5kdc[400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821067, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:17:47.446761 775 init.cc:377] Logged in from keytab as kudu/127.0.98.67@KRBTEST.COM (short username kudu)
I20260618 22:17:47.447715 775 webserver.cc:460] Webserver started at http://127.0.98.67:36097/ using document root <none> and password file <none>
I20260618 22:17:47.448277 775 fs_manager.cc:359] Metadata directory not provided
I20260618 22:17:47.448376 775 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:17:47.448618 775 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:17:47.450760 775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/minicluster-data/ts-2/data/instance:
uuid: "db4861a197594a728de3699dc09af388"
format_stamp: "Formatted at 2026-06-18 22:17:47 on dist-test-slave-g6r4"
server_key: "2cbbe8744a3186dc628834487ad0c8a6"
server_key_iv: "78d61a24157fd8977a8cd62d1a375926"
server_key_version: "encryptionkey@0"
I20260618 22:17:47.451371 775 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/minicluster-data/ts-2/wal/instance:
uuid: "db4861a197594a728de3699dc09af388"
format_stamp: "Formatted at 2026-06-18 22:17:47 on dist-test-slave-g6r4"
server_key: "2cbbe8744a3186dc628834487ad0c8a6"
server_key_iv: "78d61a24157fd8977a8cd62d1a375926"
server_key_version: "encryptionkey@0"
I20260618 22:17:47.455101 775 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260618 22:17:47.457396 790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:17:47.458685 775 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260618 22:17:47.458835 775 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/minicluster-data/ts-2/wal
uuid: "db4861a197594a728de3699dc09af388"
format_stamp: "Formatted at 2026-06-18 22:17:47 on dist-test-slave-g6r4"
server_key: "2cbbe8744a3186dc628834487ad0c8a6"
server_key_iv: "78d61a24157fd8977a8cd62d1a375926"
server_key_version: "encryptionkey@0"
I20260618 22:17:47.458949 775 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:17:47.472103 775 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:17:47.475157 775 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:17:47.475363 775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:17:47.475916 775 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:17:47.476673 775 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:17:47.476752 775 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:17:47.476815 775 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:17:47.476881 775 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:17:47.486244 775 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:41359
I20260618 22:17:47.486322 903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:41359 every 8 connection(s)
I20260618 22:17:47.487109 775 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
I20260618 22:17:47.487963 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 775
I20260618 22:17:47.488108 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781821065380026-393-0/minicluster-data/ts-2/wal/instance
I20260618 22:17:47.488361 393 external_mini_cluster.cc:1428] Setting key 0691c25e601bacf648a21e6250fae28c
Jun 18 22:17:47 dist-test-slave-g6r4 krb5kdc[400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821067, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:17:47.500739 483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:47.490046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:56871 (local address 127.0.98.126:40895)
0618 22:17:47.490167 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0618 22:17:47.490171 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:17:47.490186 (+ 15us) server_negotiation.cc:407] Connection header received
0618 22:17:47.491565 (+ 1379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:17:47.491569 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:17:47.491635 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:17:47.491728 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:17:47.493179 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:47.493827 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:47.494973 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:47.495200 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:47.495315 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:47.497531 (+ 2216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:17:47.497534 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:17:47.497536 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:17:47.497558 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:17:47.499039 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:17:47.499499 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:17:47.499502 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:17:47.499503 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:17:47.499571 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:17:47.499857 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:17:47.499859 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:17:47.499860 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:17:47.500013 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:17:47.500086 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0618 22:17:47.500529 (+ 443us) server_negotiation.cc:299] Negotiation successful
0618 22:17:47.500618 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260618 22:17:47.501480 906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:47.489327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:40895 (local address 127.0.98.67:56871)
0618 22:17:47.489767 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0618 22:17:47.489815 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0618 22:17:47.491328 (+ 1513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:17:47.491927 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:17:47.491934 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:17:47.492492 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:17:47.493001 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:47.493019 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:47.493991 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:47.493999 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:47.494842 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:47.494850 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:47.495342 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:47.495346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:47.495508 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:47.496022 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:17:47.496037 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:17:47.497371 (+ 1334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:17:47.499160 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:17:47.499164 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:17:47.499174 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:17:47.499409 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:17:47.499656 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:17:47.499659 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:17:47.499660 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:17:47.499777 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:17:47.500113 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:17:47.500118 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:17:47.500335 (+ 217us) client_negotiation.cc:769] Sending connection context
0618 22:17:47.500478 (+ 143us) client_negotiation.cc:240] Negotiation successful
0618 22:17:47.500717 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":127,"threads_started":1}
I20260618 22:17:47.502770 904 heartbeater.cc:347] Connected to a master server at 127.0.98.126:40895
I20260618 22:17:47.502979 904 heartbeater.cc:464] Registering TS with master...
I20260618 22:17:47.503392 904 heartbeater.cc:511] Master 127.0.98.126:40895 requested a full tablet report, sending...
I20260618 22:17:47.504331 445 ts_manager.cc:194] Registered new tserver with Master: db4861a197594a728de3699dc09af388 (127.0.98.67:41359)
I20260618 22:17:47.504875 445 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.67@KRBTEST.COM'} at 127.0.98.67:56871
I20260618 22:17:47.514616 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 18 22:17:47 dist-test-slave-g6r4 krb5kdc[400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821067, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 18 22:17:47 dist-test-slave-g6r4 krb5kdc[400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821067, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.65@KRBTEST.COM
I20260618 22:17:47.540453 911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:47.530688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44300 (local address 127.0.98.65:44977)
0618 22:17:47.530973 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0618 22:17:47.530988 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0618 22:17:47.531013 (+ 25us) server_negotiation.cc:407] Connection header received
0618 22:17:47.531144 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:17:47.531154 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:17:47.531351 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:17:47.531516 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:17:47.532255 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:47.533149 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:47.534131 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:47.534499 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:47.534630 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:47.536936 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:17:47.536945 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:17:47.536955 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:17:47.536978 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:17:47.538492 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:17:47.538941 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:17:47.538949 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:17:47.538955 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:17:47.539032 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:17:47.539339 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:17:47.539342 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:17:47.539344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:17:47.539572 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:17:47.539917 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0618 22:17:47.540097 (+ 180us) server_negotiation.cc:299] Negotiation successful
0618 22:17:47.540270 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":73,"threads_started":1}
W20260618 22:17:47.541416 550 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44300
Jun 18 22:17:47 dist-test-slave-g6r4 krb5kdc[400](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821067, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 18 22:17:47 dist-test-slave-g6r4 krb5kdc[400](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821067, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.65@KRBTEST.COM
I20260618 22:17:47.566320 911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:47.557465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44302 (local address 127.0.98.65:44977)
0618 22:17:47.557617 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0618 22:17:47.557622 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:17:47.557751 (+ 129us) server_negotiation.cc:407] Connection header received
0618 22:17:47.557924 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:17:47.557927 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:17:47.557977 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:17:47.558074 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:17:47.558741 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:47.559625 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:47.560502 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:47.560746 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:47.560862 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:47.563159 (+ 2297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:17:47.563162 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:17:47.563164 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:17:47.563185 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:17:47.564638 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:17:47.565050 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:17:47.565054 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:17:47.565056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:17:47.565152 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:17:47.565419 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:17:47.565422 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:17:47.565423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:17:47.565598 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:17:47.565705 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0618 22:17:47.566015 (+ 310us) server_negotiation.cc:299] Negotiation successful
0618 22:17:47.566145 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260618 22:17:47.567309 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 507
I20260618 22:17:47.573061 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 641
I20260618 22:17:47.582688 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 775
I20260618 22:17:47.587764 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 413
2026-06-18T22:17:47Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2200 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:17:47 dist-test-slave-g6r4 krb5kdc[918](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:17:47 dist-test-slave-g6r4 krb5kdc[918](info): set up 2 sockets
Jun 18 22:17:47 dist-test-slave-g6r4 krb5kdc[918](info): commencing operation
krb5kdc: starting...
W20260618 22:17:48.633482 393 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/krb5kdc/test-user.keytab.
Jun 18 22:17:48 dist-test-slave-g6r4 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:17:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:17:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260618 22:17:48.752871 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:39807
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:41397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:39807
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:17:48.862088 931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:17:48.862362 931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:17:48.862458 931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:17:48.866825 931 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:17:48.866878 931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:17:48.866897 931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:17:48.866914 931 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:17:48.866961 931 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:17:48.872267 931 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:41397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:39807
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:39807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:17:48.873258 931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:17:48.874277 931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260618 22:17:48.882485 931 server_base.cc:1034] running on GCE node
W20260618 22:17:48.882443 939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:17:48.882468 937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:17:48.882443 936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:17:48.883268 931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:17:48.884342 931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:17:48.885500 931 hybrid_clock.cc:648] HybridClock initialized: now 1781821068885480 us; error 35 us; skew 500 ppm
Jun 18 22:17:48 dist-test-slave-g6r4 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821068, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:17:48.888201 931 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260618 22:17:48.889011 931 webserver.cc:460] Webserver started at http://127.0.98.126:34845/ using document root <none> and password file <none>
I20260618 22:17:48.889554 931 fs_manager.cc:359] Metadata directory not provided
I20260618 22:17:48.889648 931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:17:48.889863 931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:17:48.891836 931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/minicluster-data/master-0/data/instance:
uuid: "56d4e9bcc10c4fe48627ac509b8aa08f"
format_stamp: "Formatted at 2026-06-18 22:17:48 on dist-test-slave-g6r4"
server_key: "3e7735885c35703db6348821731eeb1e"
server_key_iv: "249cdaef752c510abb750d91ea5fc084"
server_key_version: "encryptionkey@0"
I20260618 22:17:48.892391 931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/minicluster-data/master-0/wal/instance:
uuid: "56d4e9bcc10c4fe48627ac509b8aa08f"
format_stamp: "Formatted at 2026-06-18 22:17:48 on dist-test-slave-g6r4"
server_key: "3e7735885c35703db6348821731eeb1e"
server_key_iv: "249cdaef752c510abb750d91ea5fc084"
server_key_version: "encryptionkey@0"
I20260618 22:17:48.895949 931 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260618 22:17:48.898631 946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:17:48.900279 931 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:17:48.900409 931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/minicluster-data/master-0/wal
uuid: "56d4e9bcc10c4fe48627ac509b8aa08f"
format_stamp: "Formatted at 2026-06-18 22:17:48 on dist-test-slave-g6r4"
server_key: "3e7735885c35703db6348821731eeb1e"
server_key_iv: "249cdaef752c510abb750d91ea5fc084"
server_key_version: "encryptionkey@0"
I20260618 22:17:48.900529 931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:17:48.917083 931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:17:48.919973 931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:17:48.920120 931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:17:48.928156 931 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:39807
I20260618 22:17:48.928166 998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:39807 every 8 connection(s)
I20260618 22:17:48.929204 931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
I20260618 22:17:48.931973 999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:17:48.937242 999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56d4e9bcc10c4fe48627ac509b8aa08f: Bootstrap starting.
I20260618 22:17:48.938763 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 931
I20260618 22:17:48.938874 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/minicluster-data/master-0/wal/instance
I20260618 22:17:48.939105 393 external_mini_cluster.cc:1428] Setting key 145d1fa2761f5a179c1ea20b5934c134
I20260618 22:17:48.939877 999 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56d4e9bcc10c4fe48627ac509b8aa08f: Neither blocks nor log segments found. Creating new log.
I20260618 22:17:48.940780 999 log.cc:824] T 00000000000000000000000000000000 P 56d4e9bcc10c4fe48627ac509b8aa08f: Log is configured to *not* fsync() on all Append() calls
I20260618 22:17:48.943089 999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56d4e9bcc10c4fe48627ac509b8aa08f: No bootstrap required, opened a new log
I20260618 22:17:48.947675 999 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56d4e9bcc10c4fe48627ac509b8aa08f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d4e9bcc10c4fe48627ac509b8aa08f" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39807 } }
I20260618 22:17:48.947952 999 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56d4e9bcc10c4fe48627ac509b8aa08f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:17:48.948009 999 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56d4e9bcc10c4fe48627ac509b8aa08f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56d4e9bcc10c4fe48627ac509b8aa08f, State: Initialized, Role: FOLLOWER
I20260618 22:17:48.948501 999 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56d4e9bcc10c4fe48627ac509b8aa08f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d4e9bcc10c4fe48627ac509b8aa08f" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39807 } }
Jun 18 22:17:48 dist-test-slave-g6r4 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:17:48.948657 999 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56d4e9bcc10c4fe48627ac509b8aa08f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:17:48.948734 999 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56d4e9bcc10c4fe48627ac509b8aa08f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:17:48.948833 999 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56d4e9bcc10c4fe48627ac509b8aa08f [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:17:48.949762 999 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56d4e9bcc10c4fe48627ac509b8aa08f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d4e9bcc10c4fe48627ac509b8aa08f" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39807 } }
I20260618 22:17:48.950167 999 leader_election.cc:302] T 00000000000000000000000000000000 P 56d4e9bcc10c4fe48627ac509b8aa08f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56d4e9bcc10c4fe48627ac509b8aa08f; no voters:
I20260618 22:17:48.950557 999 leader_election.cc:288] T 00000000000000000000000000000000 P 56d4e9bcc10c4fe48627ac509b8aa08f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:17:48.950693 1004 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56d4e9bcc10c4fe48627ac509b8aa08f [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:17:48.950911 1004 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56d4e9bcc10c4fe48627ac509b8aa08f [term 1 LEADER]: Becoming Leader. State: Replica: 56d4e9bcc10c4fe48627ac509b8aa08f, State: Running, Role: LEADER
I20260618 22:17:48.951272 1004 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56d4e9bcc10c4fe48627ac509b8aa08f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d4e9bcc10c4fe48627ac509b8aa08f" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39807 } }
I20260618 22:17:48.951822 999 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56d4e9bcc10c4fe48627ac509b8aa08f [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:17:48.952927 1006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56d4e9bcc10c4fe48627ac509b8aa08f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56d4e9bcc10c4fe48627ac509b8aa08f. Latest consensus state: current_term: 1 leader_uuid: "56d4e9bcc10c4fe48627ac509b8aa08f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d4e9bcc10c4fe48627ac509b8aa08f" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39807 } } }
I20260618 22:17:48.953056 1006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56d4e9bcc10c4fe48627ac509b8aa08f [sys.catalog]: This master's current role is: LEADER
I20260618 22:17:48.953377 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56d4e9bcc10c4fe48627ac509b8aa08f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56d4e9bcc10c4fe48627ac509b8aa08f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d4e9bcc10c4fe48627ac509b8aa08f" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39807 } } }
I20260618 22:17:48.953465 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56d4e9bcc10c4fe48627ac509b8aa08f [sys.catalog]: This master's current role is: LEADER
I20260618 22:17:48.955787 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:48.940268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59570 (local address 127.0.98.126:39807)
0618 22:17:48.940687 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0618 22:17:48.940701 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0618 22:17:48.940735 (+ 34us) server_negotiation.cc:407] Connection header received
0618 22:17:48.941583 (+ 848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:17:48.941607 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:17:48.941983 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:17:48.942260 (+ 277us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:17:48.943132 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:48.944301 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:48.945492 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:48.946047 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:48.946166 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:48.949245 (+ 3079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:17:48.949250 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:17:48.949262 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:17:48.949287 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:17:48.951448 (+ 2161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:17:48.952081 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:17:48.952087 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:17:48.952093 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:17:48.952228 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:17:48.952598 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:17:48.952617 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:17:48.952618 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:17:48.953153 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:17:48.953347 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0618 22:17:48.954620 (+ 1273us) server_negotiation.cc:299] Negotiation successful
0618 22:17:48.954947 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":133,"threads_started":1}
W20260618 22:17:48.957015 1019 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 56d4e9bcc10c4fe48627ac509b8aa08f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 22:17:48.957093 1019 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 22:17:48.957857 1013 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:17:48.958618 1013 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:17:48.963985 1013 catalog_manager.cc:1347] Generated new cluster ID: 51df460966184348a6cba9ac0bfa9e0a
I20260618 22:17:48.964049 1013 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:17:48.977465 1013 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:17:48.978509 1013 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:17:48.984306 1013 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 56d4e9bcc10c4fe48627ac509b8aa08f: Generated new TSK 0
I20260618 22:17:48.984966 1013 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab.
I20260618 22:17:49.016669 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:39807
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:41397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:17:49.125442 1025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:17:49.125738 1025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:17:49.125816 1025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:17:49.129727 1025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:17:49.129815 1025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:17:49.129917 1025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260618 22:17:49.135000 1025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:41397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:39807
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:17:49.135982 1025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:17:49.136946 1025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:17:49.146235 1031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:17:49.146344 1025 server_base.cc:1034] running on GCE node
W20260618 22:17:49.146307 1030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:17:49.146229 1033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:17:49.146895 1025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:17:49.147534 1025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:17:49.148731 1025 hybrid_clock.cc:648] HybridClock initialized: now 1781821069148704 us; error 47 us; skew 500 ppm
Jun 18 22:17:49 dist-test-slave-g6r4 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821069, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:17:49.151539 1025 init.cc:377] Logged in from keytab as kudu/127.0.98.65@KRBTEST.COM (short username kudu)
I20260618 22:17:49.152613 1025 webserver.cc:460] Webserver started at http://127.0.98.65:33441/ using document root <none> and password file <none>
I20260618 22:17:49.153177 1025 fs_manager.cc:359] Metadata directory not provided
I20260618 22:17:49.153270 1025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:17:49.153468 1025 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:17:49.155476 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/minicluster-data/ts-0/data/instance:
uuid: "939d2e94e8cd4c139a603b138b7d62b4"
format_stamp: "Formatted at 2026-06-18 22:17:49 on dist-test-slave-g6r4"
server_key: "3c018d0a682c31be895bf0953bb320a8"
server_key_iv: "d68278beb859b13ffedd0b6392a74800"
server_key_version: "encryptionkey@0"
I20260618 22:17:49.156036 1025 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/minicluster-data/ts-0/wal/instance:
uuid: "939d2e94e8cd4c139a603b138b7d62b4"
format_stamp: "Formatted at 2026-06-18 22:17:49 on dist-test-slave-g6r4"
server_key: "3c018d0a682c31be895bf0953bb320a8"
server_key_iv: "d68278beb859b13ffedd0b6392a74800"
server_key_version: "encryptionkey@0"
I20260618 22:17:49.159685 1025 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260618 22:17:49.162143 1040 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:17:49.163334 1025 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260618 22:17:49.163460 1025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/minicluster-data/ts-0/wal
uuid: "939d2e94e8cd4c139a603b138b7d62b4"
format_stamp: "Formatted at 2026-06-18 22:17:49 on dist-test-slave-g6r4"
server_key: "3c018d0a682c31be895bf0953bb320a8"
server_key_iv: "d68278beb859b13ffedd0b6392a74800"
server_key_version: "encryptionkey@0"
I20260618 22:17:49.163565 1025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:17:49.187458 1025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:17:49.190845 1025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:17:49.191056 1025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:17:49.191641 1025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:17:49.192428 1025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:17:49.192474 1025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:17:49.192551 1025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:17:49.192611 1025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:17:49.201758 1025 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:42133
I20260618 22:17:49.201787 1153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:42133 every 8 connection(s)
I20260618 22:17:49.202633 1025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
I20260618 22:17:49.202921 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 1025
I20260618 22:17:49.203039 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/minicluster-data/ts-0/wal/instance
I20260618 22:17:49.203339 393 external_mini_cluster.cc:1428] Setting key 162ba72042061b94a371dabf11990a82
Jun 18 22:17:49 dist-test-slave-g6r4 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821069, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.66@KRBTEST.COM" created.
I20260618 22:17:49.216454 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:49.204472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:48885 (local address 127.0.98.126:39807)
0618 22:17:49.204608 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0618 22:17:49.204612 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:17:49.205578 (+ 966us) server_negotiation.cc:407] Connection header received
0618 22:17:49.207030 (+ 1452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:17:49.207033 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:17:49.207087 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:17:49.207165 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:17:49.208305 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:49.208838 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:49.210040 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:49.210269 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:49.210390 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:49.212711 (+ 2321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:17:49.212719 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:17:49.212722 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:17:49.212744 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:17:49.214226 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:17:49.214733 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:17:49.214740 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:17:49.214742 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:17:49.214820 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:17:49.215238 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:17:49.215243 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:17:49.215247 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:17:49.215430 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:17:49.215509 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0618 22:17:49.216131 (+ 622us) server_negotiation.cc:299] Negotiation successful
0618 22:17:49.216261 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260618 22:17:49.217286 1156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:49.204969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39807 (local address 127.0.98.65:48885)
0618 22:17:49.205431 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0618 22:17:49.205472 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0618 22:17:49.206805 (+ 1333us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:17:49.207365 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:17:49.207372 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:17:49.207712 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:17:49.208147 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:49.208159 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:49.208968 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:49.208972 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:49.209842 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:49.209850 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:49.210420 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:49.210424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:49.210591 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:49.211179 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:17:49.211194 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:17:49.212537 (+ 1343us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:17:49.214345 (+ 1808us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:17:49.214350 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:17:49.214364 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:17:49.214616 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:17:49.214947 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:17:49.214950 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:17:49.214952 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:17:49.215135 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:17:49.215544 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:17:49.215553 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:17:49.215872 (+ 319us) client_negotiation.cc:769] Sending connection context
0618 22:17:49.216087 (+ 215us) client_negotiation.cc:240] Negotiation successful
0618 22:17:49.216455 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":135,"threads_started":1}
I20260618 22:17:49.218524 1154 heartbeater.cc:347] Connected to a master server at 127.0.98.126:39807
I20260618 22:17:49.218794 1154 heartbeater.cc:464] Registering TS with master...
I20260618 22:17:49.219372 1154 heartbeater.cc:511] Master 127.0.98.126:39807 requested a full tablet report, sending...
I20260618 22:17:49.221125 963 ts_manager.cc:194] Registered new tserver with Master: 939d2e94e8cd4c139a603b138b7d62b4 (127.0.98.65:42133)
I20260618 22:17:49.222755 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.65@KRBTEST.COM'} at 127.0.98.65:48885
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab.
I20260618 22:17:49.233464 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:39807
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:41397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:17:49.340550 1159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:17:49.340804 1159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:17:49.340848 1159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:17:49.345100 1159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:17:49.345155 1159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:17:49.345218 1159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260618 22:17:49.350281 1159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:41397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:39807
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:17:49.351146 1159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:17:49.352164 1159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:17:49.361627 1167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:17:49.361624 1164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:17:49.361660 1165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:17:49.361932 1159 server_base.cc:1034] running on GCE node
I20260618 22:17:49.362394 1159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:17:49.362995 1159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:17:49.364192 1159 hybrid_clock.cc:648] HybridClock initialized: now 1781821069364172 us; error 39 us; skew 500 ppm
Jun 18 22:17:49 dist-test-slave-g6r4 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821069, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:17:49.366850 1159 init.cc:377] Logged in from keytab as kudu/127.0.98.66@KRBTEST.COM (short username kudu)
I20260618 22:17:49.367694 1159 webserver.cc:460] Webserver started at http://127.0.98.66:40893/ using document root <none> and password file <none>
I20260618 22:17:49.368230 1159 fs_manager.cc:359] Metadata directory not provided
I20260618 22:17:49.368290 1159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:17:49.368502 1159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:17:49.370476 1159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/minicluster-data/ts-1/data/instance:
uuid: "afd23b1025f94e45b42fbca01a0f8e4b"
format_stamp: "Formatted at 2026-06-18 22:17:49 on dist-test-slave-g6r4"
server_key: "6b5a87247fd1088822d8848fc7a6aefc"
server_key_iv: "bf6f10de37e095ed692f74f1374fd3a1"
server_key_version: "encryptionkey@0"
I20260618 22:17:49.371014 1159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/minicluster-data/ts-1/wal/instance:
uuid: "afd23b1025f94e45b42fbca01a0f8e4b"
format_stamp: "Formatted at 2026-06-18 22:17:49 on dist-test-slave-g6r4"
server_key: "6b5a87247fd1088822d8848fc7a6aefc"
server_key_iv: "bf6f10de37e095ed692f74f1374fd3a1"
server_key_version: "encryptionkey@0"
I20260618 22:17:49.374469 1159 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260618 22:17:49.376789 1174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:17:49.378129 1159 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:17:49.378240 1159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/minicluster-data/ts-1/wal
uuid: "afd23b1025f94e45b42fbca01a0f8e4b"
format_stamp: "Formatted at 2026-06-18 22:17:49 on dist-test-slave-g6r4"
server_key: "6b5a87247fd1088822d8848fc7a6aefc"
server_key_iv: "bf6f10de37e095ed692f74f1374fd3a1"
server_key_version: "encryptionkey@0"
I20260618 22:17:49.378381 1159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:17:49.389227 1159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:17:49.391868 1159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:17:49.392057 1159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:17:49.392594 1159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:17:49.393353 1159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:17:49.393397 1159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:17:49.393468 1159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:17:49.393508 1159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:17:49.402408 1159 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:42645
I20260618 22:17:49.402431 1287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:42645 every 8 connection(s)
I20260618 22:17:49.403262 1159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
I20260618 22:17:49.409250 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 1159
I20260618 22:17:49.409369 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/minicluster-data/ts-1/wal/instance
I20260618 22:17:49.409673 393 external_mini_cluster.cc:1428] Setting key 4170ad0e55fb22a208f2aea5ed8c84d6
Jun 18 22:17:49 dist-test-slave-g6r4 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821069, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:17:49.417754 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:49.404985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:41421 (local address 127.0.98.126:39807)
0618 22:17:49.405127 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0618 22:17:49.405132 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:17:49.406067 (+ 935us) server_negotiation.cc:407] Connection header received
0618 22:17:49.407364 (+ 1297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:17:49.407367 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:17:49.407413 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:17:49.407487 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:17:49.408558 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:49.409070 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:49.410633 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:49.410929 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:49.411033 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:49.414046 (+ 3013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:17:49.414049 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:17:49.414051 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:17:49.414072 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:17:49.415547 (+ 1475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:17:49.416216 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:17:49.416218 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:17:49.416220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:17:49.416288 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:17:49.416685 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:17:49.416687 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:17:49.416688 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:17:49.416841 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:17:49.416909 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0618 22:17:49.417546 (+ 637us) server_negotiation.cc:299] Negotiation successful
0618 22:17:49.417639 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260618 22:17:49.418915 1290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:49.405428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39807 (local address 127.0.98.66:41421)
0618 22:17:49.405876 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0618 22:17:49.405930 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0618 22:17:49.407176 (+ 1246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:17:49.407640 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:17:49.407647 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:17:49.407996 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:17:49.408438 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:49.408450 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:49.409206 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:49.409211 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:49.410513 (+ 1302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:49.410523 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:49.411083 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:49.411087 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:49.411291 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:49.411898 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:17:49.411922 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:17:49.413857 (+ 1935us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:17:49.415674 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:17:49.415683 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:17:49.415701 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:17:49.416079 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:17:49.416384 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:17:49.416388 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:17:49.416394 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:17:49.416583 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:17:49.416947 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:17:49.416952 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:17:49.417283 (+ 331us) client_negotiation.cc:769] Sending connection context
0618 22:17:49.417490 (+ 207us) client_negotiation.cc:240] Negotiation successful
0618 22:17:49.417832 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":104,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.67@KRBTEST.COM; defaulting to no policy
I20260618 22:17:49.420131 1288 heartbeater.cc:347] Connected to a master server at 127.0.98.126:39807
I20260618 22:17:49.420473 1288 heartbeater.cc:464] Registering TS with master...
I20260618 22:17:49.420907 1288 heartbeater.cc:511] Master 127.0.98.126:39807 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.67@KRBTEST.COM" created.
I20260618 22:17:49.422207 963 ts_manager.cc:194] Registered new tserver with Master: afd23b1025f94e45b42fbca01a0f8e4b (127.0.98.66:42645)
I20260618 22:17:49.422914 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.66@KRBTEST.COM'} at 127.0.98.66:41421
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab.
I20260618 22:17:49.438275 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:39807
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:41397
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:17:49.545069 1293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:17:49.545320 1293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:17:49.545393 1293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:17:49.549335 1293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:17:49.549419 1293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:17:49.549506 1293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260618 22:17:49.554665 1293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:41397
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:39807
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:17:49.555583 1293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:17:49.556543 1293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:17:49.565578 1301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:17:49.565661 1293 server_base.cc:1034] running on GCE node
W20260618 22:17:49.565569 1298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:17:49.565587 1299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:17:49.566205 1293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:17:49.566896 1293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:17:49.568112 1293 hybrid_clock.cc:648] HybridClock initialized: now 1781821069568076 us; error 44 us; skew 500 ppm
Jun 18 22:17:49 dist-test-slave-g6r4 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821069, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:17:49.570796 1293 init.cc:377] Logged in from keytab as kudu/127.0.98.67@KRBTEST.COM (short username kudu)
I20260618 22:17:49.571516 1293 webserver.cc:460] Webserver started at http://127.0.98.67:45557/ using document root <none> and password file <none>
I20260618 22:17:49.571983 1293 fs_manager.cc:359] Metadata directory not provided
I20260618 22:17:49.572037 1293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:17:49.572183 1293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:17:49.574074 1293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/minicluster-data/ts-2/data/instance:
uuid: "360db1b5dfd04fffa59dafa22ad8c442"
format_stamp: "Formatted at 2026-06-18 22:17:49 on dist-test-slave-g6r4"
server_key: "a8f081fd556cd6c76b12dff63f4f6f5b"
server_key_iv: "127bf9289341428d880cd811d577e4b0"
server_key_version: "encryptionkey@0"
I20260618 22:17:49.574553 1293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/minicluster-data/ts-2/wal/instance:
uuid: "360db1b5dfd04fffa59dafa22ad8c442"
format_stamp: "Formatted at 2026-06-18 22:17:49 on dist-test-slave-g6r4"
server_key: "a8f081fd556cd6c76b12dff63f4f6f5b"
server_key_iv: "127bf9289341428d880cd811d577e4b0"
server_key_version: "encryptionkey@0"
I20260618 22:17:49.577843 1293 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.001s
I20260618 22:17:49.580058 1308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:17:49.581205 1293 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:17:49.581374 1293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/minicluster-data/ts-2/wal
uuid: "360db1b5dfd04fffa59dafa22ad8c442"
format_stamp: "Formatted at 2026-06-18 22:17:49 on dist-test-slave-g6r4"
server_key: "a8f081fd556cd6c76b12dff63f4f6f5b"
server_key_iv: "127bf9289341428d880cd811d577e4b0"
server_key_version: "encryptionkey@0"
I20260618 22:17:49.581468 1293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:17:49.598852 1293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:17:49.601804 1293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:17:49.602027 1293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:17:49.602584 1293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:17:49.603358 1293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:17:49.603401 1293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:17:49.603466 1293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:17:49.603497 1293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:17:49.611994 1293 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:46387
I20260618 22:17:49.612046 1421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:46387 every 8 connection(s)
I20260618 22:17:49.612869 1293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
I20260618 22:17:49.613600 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 1293
I20260618 22:17:49.613742 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781821065380026-393-0/minicluster-data/ts-2/wal/instance
I20260618 22:17:49.614100 393 external_mini_cluster.cc:1428] Setting key 82daabd77f46fced4138f5dc15654571
Jun 18 22:17:49 dist-test-slave-g6r4 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821069, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:17:49.626264 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:49.614802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:37039 (local address 127.0.98.126:39807)
0618 22:17:49.614933 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0618 22:17:49.614937 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:17:49.616395 (+ 1458us) server_negotiation.cc:407] Connection header received
0618 22:17:49.617594 (+ 1199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:17:49.617596 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:17:49.617638 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:17:49.617705 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:17:49.618750 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:49.619264 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:49.620393 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:49.620598 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:49.620672 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:49.622917 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:17:49.622921 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:17:49.622923 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:17:49.622944 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:17:49.624421 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:17:49.624931 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:17:49.624933 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:17:49.624934 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:17:49.625005 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:17:49.625306 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:17:49.625308 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:17:49.625309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:17:49.625463 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:17:49.625534 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0618 22:17:49.626033 (+ 499us) server_negotiation.cc:299] Negotiation successful
0618 22:17:49.626145 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260618 22:17:49.627205 1424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:49.615297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39807 (local address 127.0.98.67:37039)
0618 22:17:49.616244 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0618 22:17:49.616301 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0618 22:17:49.617423 (+ 1122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:17:49.617829 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:17:49.617835 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:17:49.618183 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:17:49.618631 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:49.618644 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:49.619366 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:49.619368 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:49.620284 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:49.620293 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:49.620716 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:49.620718 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:49.620850 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:49.621347 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:17:49.621361 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:17:49.622776 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:17:49.624530 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:17:49.624534 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:17:49.624543 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:17:49.624808 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:17:49.625090 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:17:49.625093 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:17:49.625094 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:17:49.625224 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:17:49.625547 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:17:49.625552 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:17:49.625777 (+ 225us) client_negotiation.cc:769] Sending connection context
0618 22:17:49.625997 (+ 220us) client_negotiation.cc:240] Negotiation successful
0618 22:17:49.626229 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"thread_start_us":117,"threads_started":1}
I20260618 22:17:49.628455 1422 heartbeater.cc:347] Connected to a master server at 127.0.98.126:39807
I20260618 22:17:49.628690 1422 heartbeater.cc:464] Registering TS with master...
I20260618 22:17:49.629132 1422 heartbeater.cc:511] Master 127.0.98.126:39807 requested a full tablet report, sending...
I20260618 22:17:49.630120 963 ts_manager.cc:194] Registered new tserver with Master: 360db1b5dfd04fffa59dafa22ad8c442 (127.0.98.67:46387)
I20260618 22:17:49.630656 963 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.67@KRBTEST.COM'} at 127.0.98.67:37039
I20260618 22:17:49.639523 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 22:17:49.650038 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:49.642774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59584 (local address 127.0.98.126:39807)
0618 22:17:49.642936 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0618 22:17:49.642940 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:17:49.643086 (+ 146us) server_negotiation.cc:407] Connection header received
0618 22:17:49.643239 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:17:49.643245 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:17:49.643290 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:17:49.643386 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:17:49.644050 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:49.644554 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:49.645525 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:49.645766 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:49.645883 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:49.647145 (+ 1262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:17:49.647149 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:17:49.647152 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:17:49.647176 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:17:49.648308 (+ 1132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:17:49.648758 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:17:49.648762 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:17:49.648764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:17:49.648846 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:17:49.649124 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:17:49.649129 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:17:49.649130 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:17:49.649344 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:17:49.649494 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0618 22:17:49.649719 (+ 225us) server_negotiation.cc:299] Negotiation successful
0618 22:17:49.649841 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260618 22:17:49.654870 963 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59584:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 22:17:49.657747 963 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:17:49.672173 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:49.666592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49132 (local address 127.0.98.66:42645)
0618 22:17:49.666944 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0618 22:17:49.666954 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0618 22:17:49.666972 (+ 18us) server_negotiation.cc:407] Connection header received
0618 22:17:49.667032 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:17:49.667036 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:17:49.667247 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:17:49.667506 (+ 259us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:17:49.667741 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:49.668970 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:49.670780 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:49.671508 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:49.671613 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:49.671733 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0618 22:17:49.671985 (+ 252us) server_negotiation.cc:299] Negotiation successful
0618 22:17:49.672056 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":146,"threads_started":1}
I20260618 22:17:49.672187 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:49.666120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:42645 (local address 127.0.0.1:49132)
0618 22:17:49.666352 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0618 22:17:49.666367 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:17:49.666461 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:17:49.667381 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:17:49.667383 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:17:49.667396 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:17:49.667606 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:49.667614 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:49.669299 (+ 1685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:49.669301 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:49.670646 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:49.670657 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:49.671622 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:49.671626 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:49.671787 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:49.671829 (+ 42us) client_negotiation.cc:769] Sending connection context
0618 22:17:49.671942 (+ 113us) client_negotiation.cc:240] Negotiation successful
0618 22:17:49.672047 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":97,"thread_start_us":82,"threads_started":1}
I20260618 22:17:49.672739 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:49.666506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:46387 (local address 127.0.0.1:53406)
0618 22:17:49.667165 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0618 22:17:49.667178 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:17:49.667384 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:17:49.668051 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:17:49.668054 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:17:49.668071 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:17:49.668161 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:49.668167 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:49.669119 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:49.669125 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:49.670486 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:49.670495 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:49.671622 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:49.671626 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:49.672412 (+ 786us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:49.672437 (+ 25us) client_negotiation.cc:769] Sending connection context
0618 22:17:49.672517 (+ 80us) client_negotiation.cc:240] Negotiation successful
0618 22:17:49.672603 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":605,"mutex_wait_us":784,"thread_start_us":82,"threads_started":1}
I20260618 22:17:49.673022 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:49.665851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:42133 (local address 127.0.0.1:34158)
0618 22:17:49.666154 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0618 22:17:49.666197 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0618 22:17:49.666322 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:17:49.667305 (+ 983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:17:49.667314 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:17:49.667344 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:17:49.667542 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:49.667553 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:49.669178 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:49.669181 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:49.670655 (+ 1474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:49.670661 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:49.672618 (+ 1957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:49.672621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:49.672723 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:49.672745 (+ 22us) client_negotiation.cc:769] Sending connection context
0618 22:17:49.672800 (+ 55us) client_negotiation.cc:240] Negotiation successful
0618 22:17:49.672867 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":243,"spinlock_wait_cycles":6016,"thread_start_us":80,"threads_started":1}
I20260618 22:17:49.673328 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:49.666807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53406 (local address 127.0.98.67:46387)
0618 22:17:49.667612 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0618 22:17:49.667623 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0618 22:17:49.667645 (+ 22us) server_negotiation.cc:407] Connection header received
0618 22:17:49.667718 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:17:49.667723 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:17:49.667920 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:17:49.668076 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:17:49.668241 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:49.668976 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:49.670628 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:49.671473 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:49.671596 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:49.671733 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0618 22:17:49.673048 (+ 1315us) server_negotiation.cc:299] Negotiation successful
0618 22:17:49.673159 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":712,"thread_start_us":102,"threads_started":1}
I20260618 22:17:49.673738 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:49.666008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34158 (local address 127.0.98.65:42133)
0618 22:17:49.666782 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0618 22:17:49.666801 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0618 22:17:49.666828 (+ 27us) server_negotiation.cc:407] Connection header received
0618 22:17:49.666902 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:17:49.666908 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:17:49.667106 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:17:49.667681 (+ 575us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:17:49.667843 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:49.668960 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:49.671686 (+ 2726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:49.672498 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:49.673354 (+ 856us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:49.673508 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0618 22:17:49.673563 (+ 55us) server_negotiation.cc:299] Negotiation successful
0618 22:17:49.673629 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":89,"threads_started":1}
I20260618 22:17:49.675706 1356 tablet_service.cc:1467] Processing CreateTablet for tablet f09b2e2d8288444eb9acf9c2fc1e8ac3 (DEFAULT_TABLE table=test-table [id=861380fbbeea41c08b7f769e2faa6fc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:17:49.675714 1222 tablet_service.cc:1467] Processing CreateTablet for tablet f09b2e2d8288444eb9acf9c2fc1e8ac3 (DEFAULT_TABLE table=test-table [id=861380fbbeea41c08b7f769e2faa6fc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:17:49.676195 1088 tablet_service.cc:1467] Processing CreateTablet for tablet f09b2e2d8288444eb9acf9c2fc1e8ac3 (DEFAULT_TABLE table=test-table [id=861380fbbeea41c08b7f769e2faa6fc7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:17:49.676692 1222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f09b2e2d8288444eb9acf9c2fc1e8ac3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:17:49.676695 1356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f09b2e2d8288444eb9acf9c2fc1e8ac3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:17:49.677035 1088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f09b2e2d8288444eb9acf9c2fc1e8ac3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:17:49.682781 1439 tablet_bootstrap.cc:492] T f09b2e2d8288444eb9acf9c2fc1e8ac3 P 360db1b5dfd04fffa59dafa22ad8c442: Bootstrap starting.
I20260618 22:17:49.684002 1441 tablet_bootstrap.cc:492] T f09b2e2d8288444eb9acf9c2fc1e8ac3 P 939d2e94e8cd4c139a603b138b7d62b4: Bootstrap starting.
I20260618 22:17:49.684958 1439 tablet_bootstrap.cc:654] T f09b2e2d8288444eb9acf9c2fc1e8ac3 P 360db1b5dfd04fffa59dafa22ad8c442: Neither blocks nor log segments found. Creating new log.
I20260618 22:17:49.685797 1441 tablet_bootstrap.cc:654] T f09b2e2d8288444eb9acf9c2fc1e8ac3 P 939d2e94e8cd4c139a603b138b7d62b4: Neither blocks nor log segments found. Creating new log.
I20260618 22:17:49.686273 1439 log.cc:824] T f09b2e2d8288444eb9acf9c2fc1e8ac3 P 360db1b5dfd04fffa59dafa22ad8c442: Log is configured to *not* fsync() on all Append() calls
I20260618 22:17:49.686892 1441 log.cc:824] T f09b2e2d8288444eb9acf9c2fc1e8ac3 P 939d2e94e8cd4c139a603b138b7d62b4: Log is configured to *not* fsync() on all Append() calls
I20260618 22:17:49.686898 1440 tablet_bootstrap.cc:492] T f09b2e2d8288444eb9acf9c2fc1e8ac3 P afd23b1025f94e45b42fbca01a0f8e4b: Bootstrap starting.
I20260618 22:17:49.688077 1439 tablet_bootstrap.cc:492] T f09b2e2d8288444eb9acf9c2fc1e8ac3 P 360db1b5dfd04fffa59dafa22ad8c442: No bootstrap required, opened a new log
I20260618 22:17:49.688277 1439 ts_tablet_manager.cc:1397] T f09b2e2d8288444eb9acf9c2fc1e8ac3 P 360db1b5dfd04fffa59dafa22ad8c442: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260618 22:17:49.688804 1441 tablet_bootstrap.cc:492] T f09b2e2d8288444eb9acf9c2fc1e8ac3 P 939d2e94e8cd4c139a603b138b7d62b4: No bootstrap required, opened a new log
I20260618 22:17:49.689000 1441 ts_tablet_manager.cc:1397] T f09b2e2d8288444eb9acf9c2fc1e8ac3 P 939d2e94e8cd4c139a603b138b7d62b4: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260618 22:17:49.689291 1440 tablet_bootstrap.cc:654] T f09b2e2d8288444eb9acf9c2fc1e8ac3 P afd23b1025f94e45b42fbca01a0f8e4b: Neither blocks nor log segments found. Creating new log.
I20260618 22:17:49.690078 1440 log.cc:824] T f09b2e2d8288444eb9acf9c2fc1e8ac3 P afd23b1025f94e45b42fbca01a0f8e4b: Log is configured to *not* fsync() on all Append() calls
I20260618 22:17:49.691376 1439 raft_consensus.cc:348] T f09b2e2d8288444eb9acf9c2fc1e8ac3 P 360db1b5dfd04fffa59dafa22ad8c442 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "939d2e94e8cd4c139a603b138b7d62b4" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 42133 } } peers { permanent_uuid: "360db1b5dfd04fffa59dafa22ad8c442" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46387 } } peers { permanent_uuid: "afd23b1025f94e45b42fbca01a0f8e4b" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 42645 } }
I20260618 22:17:49.691682 1439 raft_consensus.cc:374] T f09b2e2d8288444eb9acf9c2fc1e8ac3 P 360db1b5dfd04fffa59dafa22ad8c442 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:17:49.691771 1439 raft_consensus.cc:729] T f09b2e2d8288444eb9acf9c2fc1e8ac3 P 360db1b5dfd04fffa59dafa22ad8c442 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 360db1b5dfd04fffa59dafa22ad8c442, State: Initialized, Role: FOLLOWER
I20260618 22:17:49.692210 1439 consensus_queue.cc:260] T f09b2e2d8288444eb9acf9c2fc1e8ac3 P 360db1b5dfd04fffa59dafa22ad8c442 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "939d2e94e8cd4c139a603b138b7d62b4" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 42133 } } peers { permanent_uuid: "360db1b5dfd04fffa59dafa22ad8c442" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46387 } } peers { permanent_uuid: "afd23b1025f94e45b42fbca01a0f8e4b" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 42645 } }
I20260618 22:17:49.692341 1440 tablet_bootstrap.cc:492] T f09b2e2d8288444eb9acf9c2fc1e8ac3 P afd23b1025f94e45b42fbca01a0f8e4b: No bootstrap required, opened a new log
I20260618 22:17:49.692534 1440 ts_tablet_manager.cc:1397] T f09b2e2d8288444eb9acf9c2fc1e8ac3 P afd23b1025f94e45b42fbca01a0f8e4b: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260618 22:17:49.692790 1422 heartbeater.cc:503] Master 127.0.98.126:39807 was elected leader, sending a full tablet report...
I20260618 22:17:49.693136 1439 ts_tablet_manager.cc:1428] T f09b2e2d8288444eb9acf9c2fc1e8ac3 P 360db1b5dfd04fffa59dafa22ad8c442: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260618 22:17:49.693048 1441 raft_consensus.cc:348] T f09b2e2d8288444eb9acf9c2fc1e8ac3 P 939d2e94e8cd4c139a603b138b7d62b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "939d2e94e8cd4c139a603b138b7d62b4" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 42133 } } peers { permanent_uuid: "360db1b5dfd04fffa59dafa22ad8c442" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46387 } } peers { permanent_uuid: "afd23b1025f94e45b42fbca01a0f8e4b" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 42645 } }
I20260618 22:17:49.693284 1441 raft_consensus.cc:374] T f09b2e2d8288444eb9acf9c2fc1e8ac3 P 939d2e94e8cd4c139a603b138b7d62b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:17:49.693332 1441 raft_consensus.cc:729] T f09b2e2d8288444eb9acf9c2fc1e8ac3 P 939d2e94e8cd4c139a603b138b7d62b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 939d2e94e8cd4c139a603b138b7d62b4, State: Initialized, Role: FOLLOWER
I20260618 22:17:49.694150 1441 consensus_queue.cc:260] T f09b2e2d8288444eb9acf9c2fc1e8ac3 P 939d2e94e8cd4c139a603b138b7d62b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "939d2e94e8cd4c139a603b138b7d62b4" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 42133 } } peers { permanent_uuid: "360db1b5dfd04fffa59dafa22ad8c442" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46387 } } peers { permanent_uuid: "afd23b1025f94e45b42fbca01a0f8e4b" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 42645 } }
I20260618 22:17:49.694818 1154 heartbeater.cc:503] Master 127.0.98.126:39807 was elected leader, sending a full tablet report...
I20260618 22:17:49.695160 1441 ts_tablet_manager.cc:1428] T f09b2e2d8288444eb9acf9c2fc1e8ac3 P 939d2e94e8cd4c139a603b138b7d62b4: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260618 22:17:49.696112 1440 raft_consensus.cc:348] T f09b2e2d8288444eb9acf9c2fc1e8ac3 P afd23b1025f94e45b42fbca01a0f8e4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "939d2e94e8cd4c139a603b138b7d62b4" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 42133 } } peers { permanent_uuid: "360db1b5dfd04fffa59dafa22ad8c442" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46387 } } peers { permanent_uuid: "afd23b1025f94e45b42fbca01a0f8e4b" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 42645 } }
I20260618 22:17:49.696308 1440 raft_consensus.cc:374] T f09b2e2d8288444eb9acf9c2fc1e8ac3 P afd23b1025f94e45b42fbca01a0f8e4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:17:49.696341 1440 raft_consensus.cc:729] T f09b2e2d8288444eb9acf9c2fc1e8ac3 P afd23b1025f94e45b42fbca01a0f8e4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: afd23b1025f94e45b42fbca01a0f8e4b, State: Initialized, Role: FOLLOWER
I20260618 22:17:49.696777 1440 consensus_queue.cc:260] T f09b2e2d8288444eb9acf9c2fc1e8ac3 P afd23b1025f94e45b42fbca01a0f8e4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "939d2e94e8cd4c139a603b138b7d62b4" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 42133 } } peers { permanent_uuid: "360db1b5dfd04fffa59dafa22ad8c442" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46387 } } peers { permanent_uuid: "afd23b1025f94e45b42fbca01a0f8e4b" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 42645 } }
I20260618 22:17:49.697515 1288 heartbeater.cc:503] Master 127.0.98.126:39807 was elected leader, sending a full tablet report...
I20260618 22:17:49.697702 1440 ts_tablet_manager.cc:1428] T f09b2e2d8288444eb9acf9c2fc1e8ac3 P afd23b1025f94e45b42fbca01a0f8e4b: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
W20260618 22:17:49.703764 1155 tablet.cc:2369] T f09b2e2d8288444eb9acf9c2fc1e8ac3 P 939d2e94e8cd4c139a603b138b7d62b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260618 22:17:49.863849 1423 tablet.cc:2369] T f09b2e2d8288444eb9acf9c2fc1e8ac3 P 360db1b5dfd04fffa59dafa22ad8c442: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260618 22:17:49.904389 1289 tablet.cc:2369] T f09b2e2d8288444eb9acf9c2fc1e8ac3 P afd23b1025f94e45b42fbca01a0f8e4b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:17:49.925609 1447 raft_consensus.cc:482] T f09b2e2d8288444eb9acf9c2fc1e8ac3 P afd23b1025f94e45b42fbca01a0f8e4b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:17:49.925863 1447 raft_consensus.cc:504] T f09b2e2d8288444eb9acf9c2fc1e8ac3 P afd23b1025f94e45b42fbca01a0f8e4b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "939d2e94e8cd4c139a603b138b7d62b4" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 42133 } } peers { permanent_uuid: "360db1b5dfd04fffa59dafa22ad8c442" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46387 } } peers { permanent_uuid: "afd23b1025f94e45b42fbca01a0f8e4b" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 42645 } }
I20260618 22:17:49.927273 1447 leader_election.cc:288] T f09b2e2d8288444eb9acf9c2fc1e8ac3 P afd23b1025f94e45b42fbca01a0f8e4b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 939d2e94e8cd4c139a603b138b7d62b4 (127.0.98.65:42133), 360db1b5dfd04fffa59dafa22ad8c442 (127.0.98.67:46387)
I20260618 22:17:49.931627 1448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:49.927600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:42133 (local address 127.0.98.66:54223)
0618 22:17:49.927866 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0618 22:17:49.927883 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0618 22:17:49.927991 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:17:49.928410 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:17:49.928413 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:17:49.928432 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:17:49.928561 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:49.928568 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:49.929621 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:49.929624 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:49.930568 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:49.930577 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:49.931217 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:49.931220 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:49.931326 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:49.931350 (+ 24us) client_negotiation.cc:769] Sending connection context
0618 22:17:49.931420 (+ 70us) client_negotiation.cc:240] Negotiation successful
0618 22:17:49.931482 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":22,"thread_start_us":110,"threads_started":1}
I20260618 22:17:49.931649 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:49.927643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:54223 (local address 127.0.98.65:42133)
0618 22:17:49.927765 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0618 22:17:49.927769 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:17:49.928017 (+ 248us) server_negotiation.cc:407] Connection header received
0618 22:17:49.928132 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:17:49.928138 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:17:49.928199 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:17:49.928252 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:17:49.928694 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:49.929482 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:49.930696 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:49.931105 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:49.931211 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:49.931277 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0618 22:17:49.931450 (+ 173us) server_negotiation.cc:299] Negotiation successful
0618 22:17:49.931517 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260618 22:17:49.932348 1108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f09b2e2d8288444eb9acf9c2fc1e8ac3" candidate_uuid: "afd23b1025f94e45b42fbca01a0f8e4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "939d2e94e8cd4c139a603b138b7d62b4" is_pre_election: true
I20260618 22:17:49.932457 1449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:49.927975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:46387 (local address 127.0.98.66:34675)
0618 22:17:49.928190 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0618 22:17:49.928204 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:17:49.928322 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:17:49.928657 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:17:49.928660 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:17:49.928676 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:17:49.928803 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:49.928810 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:49.929733 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:49.929737 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:49.930978 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:49.930988 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:49.932041 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:49.932045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:49.932144 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:49.932168 (+ 24us) client_negotiation.cc:769] Sending connection context
0618 22:17:49.932257 (+ 89us) client_negotiation.cc:240] Negotiation successful
0618 22:17:49.932319 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":1,"thread_start_us":62,"threads_started":1}
I20260618 22:17:49.932466 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:49.928315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:34675 (local address 127.0.98.67:46387)
0618 22:17:49.928435 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0618 22:17:49.928439 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:17:49.928451 (+ 12us) server_negotiation.cc:407] Connection header received
0618 22:17:49.928495 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:17:49.928498 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:17:49.928560 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:17:49.928640 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:17:49.928930 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:49.929628 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:49.931103 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:49.931696 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:49.931805 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:49.931870 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0618 22:17:49.932273 (+ 403us) server_negotiation.cc:299] Negotiation successful
0618 22:17:49.932352 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260618 22:17:49.932648 1108 raft_consensus.cc:2454] T f09b2e2d8288444eb9acf9c2fc1e8ac3 P 939d2e94e8cd4c139a603b138b7d62b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate afd23b1025f94e45b42fbca01a0f8e4b in term 0.
I20260618 22:17:49.933001 1376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f09b2e2d8288444eb9acf9c2fc1e8ac3" candidate_uuid: "afd23b1025f94e45b42fbca01a0f8e4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "360db1b5dfd04fffa59dafa22ad8c442" is_pre_election: true
I20260618 22:17:49.933189 1175 leader_election.cc:302] T f09b2e2d8288444eb9acf9c2fc1e8ac3 P afd23b1025f94e45b42fbca01a0f8e4b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 939d2e94e8cd4c139a603b138b7d62b4, afd23b1025f94e45b42fbca01a0f8e4b; no voters:
I20260618 22:17:49.933292 1376 raft_consensus.cc:2454] T f09b2e2d8288444eb9acf9c2fc1e8ac3 P 360db1b5dfd04fffa59dafa22ad8c442 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate afd23b1025f94e45b42fbca01a0f8e4b in term 0.
I20260618 22:17:49.933480 1447 raft_consensus.cc:2781] T f09b2e2d8288444eb9acf9c2fc1e8ac3 P afd23b1025f94e45b42fbca01a0f8e4b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:17:49.933576 1447 raft_consensus.cc:482] T f09b2e2d8288444eb9acf9c2fc1e8ac3 P afd23b1025f94e45b42fbca01a0f8e4b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:17:49.933631 1447 raft_consensus.cc:3037] T f09b2e2d8288444eb9acf9c2fc1e8ac3 P afd23b1025f94e45b42fbca01a0f8e4b [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:17:49.934660 1447 raft_consensus.cc:504] T f09b2e2d8288444eb9acf9c2fc1e8ac3 P afd23b1025f94e45b42fbca01a0f8e4b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "939d2e94e8cd4c139a603b138b7d62b4" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 42133 } } peers { permanent_uuid: "360db1b5dfd04fffa59dafa22ad8c442" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46387 } } peers { permanent_uuid: "afd23b1025f94e45b42fbca01a0f8e4b" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 42645 } }
I20260618 22:17:49.934916 1447 leader_election.cc:288] T f09b2e2d8288444eb9acf9c2fc1e8ac3 P afd23b1025f94e45b42fbca01a0f8e4b [CANDIDATE]: Term 1 election: Requested vote from peers 939d2e94e8cd4c139a603b138b7d62b4 (127.0.98.65:42133), 360db1b5dfd04fffa59dafa22ad8c442 (127.0.98.67:46387)
I20260618 22:17:49.935262 1108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f09b2e2d8288444eb9acf9c2fc1e8ac3" candidate_uuid: "afd23b1025f94e45b42fbca01a0f8e4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "939d2e94e8cd4c139a603b138b7d62b4"
I20260618 22:17:49.935313 1376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f09b2e2d8288444eb9acf9c2fc1e8ac3" candidate_uuid: "afd23b1025f94e45b42fbca01a0f8e4b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "360db1b5dfd04fffa59dafa22ad8c442"
I20260618 22:17:49.935387 1108 raft_consensus.cc:3037] T f09b2e2d8288444eb9acf9c2fc1e8ac3 P 939d2e94e8cd4c139a603b138b7d62b4 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:17:49.935412 1376 raft_consensus.cc:3037] T f09b2e2d8288444eb9acf9c2fc1e8ac3 P 360db1b5dfd04fffa59dafa22ad8c442 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:17:49.936584 1108 raft_consensus.cc:2454] T f09b2e2d8288444eb9acf9c2fc1e8ac3 P 939d2e94e8cd4c139a603b138b7d62b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afd23b1025f94e45b42fbca01a0f8e4b in term 1.
I20260618 22:17:49.936578 1376 raft_consensus.cc:2454] T f09b2e2d8288444eb9acf9c2fc1e8ac3 P 360db1b5dfd04fffa59dafa22ad8c442 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate afd23b1025f94e45b42fbca01a0f8e4b in term 1.
I20260618 22:17:49.936862 1175 leader_election.cc:302] T f09b2e2d8288444eb9acf9c2fc1e8ac3 P afd23b1025f94e45b42fbca01a0f8e4b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 360db1b5dfd04fffa59dafa22ad8c442, afd23b1025f94e45b42fbca01a0f8e4b; no voters:
I20260618 22:17:49.937018 1447 raft_consensus.cc:2781] T f09b2e2d8288444eb9acf9c2fc1e8ac3 P afd23b1025f94e45b42fbca01a0f8e4b [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:17:49.937402 1447 raft_consensus.cc:686] T f09b2e2d8288444eb9acf9c2fc1e8ac3 P afd23b1025f94e45b42fbca01a0f8e4b [term 1 LEADER]: Becoming Leader. State: Replica: afd23b1025f94e45b42fbca01a0f8e4b, State: Running, Role: LEADER
I20260618 22:17:49.937765 1447 consensus_queue.cc:237] T f09b2e2d8288444eb9acf9c2fc1e8ac3 P afd23b1025f94e45b42fbca01a0f8e4b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "939d2e94e8cd4c139a603b138b7d62b4" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 42133 } } peers { permanent_uuid: "360db1b5dfd04fffa59dafa22ad8c442" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46387 } } peers { permanent_uuid: "afd23b1025f94e45b42fbca01a0f8e4b" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 42645 } }
I20260618 22:17:49.941397 963 catalog_manager.cc:5495] T f09b2e2d8288444eb9acf9c2fc1e8ac3 P afd23b1025f94e45b42fbca01a0f8e4b reported cstate change: term changed from 0 to 1, leader changed from <none> to afd23b1025f94e45b42fbca01a0f8e4b (127.0.98.66). New cstate: current_term: 1 leader_uuid: "afd23b1025f94e45b42fbca01a0f8e4b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "939d2e94e8cd4c139a603b138b7d62b4" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 42133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "360db1b5dfd04fffa59dafa22ad8c442" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "afd23b1025f94e45b42fbca01a0f8e4b" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 42645 } health_report { overall_health: HEALTHY } } }
Jun 18 22:17:49 dist-test-slave-g6r4 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.65@KRBTEST.COM
I20260618 22:17:49.963546 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:49.954415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34172 (local address 127.0.98.65:42133)
0618 22:17:49.954535 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0618 22:17:49.954540 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:17:49.954718 (+ 178us) server_negotiation.cc:407] Connection header received
0618 22:17:49.954898 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:17:49.954900 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:17:49.954940 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:17:49.955031 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:17:49.955698 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:49.956190 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:49.957018 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:49.957230 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:49.957333 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:49.959943 (+ 2610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:17:49.959953 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:17:49.959966 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:17:49.959986 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:17:49.961602 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:17:49.962122 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:17:49.962127 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:17:49.962128 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:17:49.962211 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:17:49.962592 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:17:49.962595 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:17:49.962596 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:17:49.962865 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:17:49.963023 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0618 22:17:49.963233 (+ 210us) server_negotiation.cc:299] Negotiation successful
0618 22:17:49.963356 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
Jun 18 22:17:49 dist-test-slave-g6r4 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821069, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 18 22:17:49 dist-test-slave-g6r4 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821069, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.65@KRBTEST.COM
I20260618 22:17:49.988646 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:49.980251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34188 (local address 127.0.98.65:42133)
0618 22:17:49.980431 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0618 22:17:49.980437 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:17:49.980604 (+ 167us) server_negotiation.cc:407] Connection header received
0618 22:17:49.980768 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:17:49.980771 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:17:49.980818 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:17:49.980901 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:17:49.981587 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:49.982145 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:49.982943 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:49.983166 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:49.983280 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:49.985593 (+ 2313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:17:49.985596 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:17:49.985598 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:17:49.985619 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:17:49.987058 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:17:49.987458 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:17:49.987463 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:17:49.987465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:17:49.987539 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:17:49.987806 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:17:49.987810 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:17:49.987812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:17:49.987960 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:17:49.988096 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0618 22:17:49.988355 (+ 259us) server_negotiation.cc:299] Negotiation successful
0618 22:17:49.988471 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
Jun 18 22:17:49 dist-test-slave-g6r4 krb5kdc[918](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 18 22:17:50 dist-test-slave-g6r4 krb5kdc[918](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.65@KRBTEST.COM
I20260618 22:17:50.013680 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:50.005350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34190 (local address 127.0.98.65:42133)
0618 22:17:50.005503 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0618 22:17:50.005508 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:17:50.005634 (+ 126us) server_negotiation.cc:407] Connection header received
0618 22:17:50.005778 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:17:50.005781 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:17:50.005821 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:17:50.005977 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:17:50.006721 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:50.007241 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:50.008067 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:50.008288 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:50.008389 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:50.010778 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:17:50.010782 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:17:50.010783 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:17:50.010803 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:17:50.012266 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:17:50.012641 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:17:50.012644 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:17:50.012645 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:17:50.012726 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:17:50.012957 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:17:50.012959 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:17:50.012960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:17:50.013117 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:17:50.013218 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0618 22:17:50.013397 (+ 179us) server_negotiation.cc:299] Negotiation successful
0618 22:17:50.013497 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260618 22:17:50.015067 1068 tablet_service.cc:2908] Rejecting scan request for tablet f09b2e2d8288444eb9acf9c2fc1e8ac3: Uninitialized: clean time has not yet been initialized
I20260618 22:17:50.017041 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 1025
I20260618 22:17:50.022508 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 1159
I20260618 22:17:50.027917 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 1293
I20260618 22:17:50.032976 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 931
2026-06-18T22:17:50Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2445 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:17:50 dist-test-slave-g6r4 krb5kdc[1464](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:17:50 dist-test-slave-g6r4 krb5kdc[1464](info): set up 2 sockets
Jun 18 22:17:50 dist-test-slave-g6r4 krb5kdc[1464](info): commencing operation
krb5kdc: starting...
W20260618 22:17:51.065548 393 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/krb5kdc/test-user.keytab.
Jun 18 22:17:51 dist-test-slave-g6r4 krb5kdc[1464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:17:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:17:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260618 22:17:51.176838 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:40163
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:38803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:40163
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:17:51.280376 1477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:17:51.280664 1477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:17:51.280745 1477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:17:51.284686 1477 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:17:51.284772 1477 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260618 22:17:51.284804 1477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:17:51.284847 1477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:17:51.284881 1477 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:17:51.284916 1477 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:17:51.290021 1477 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:38803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:40163
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:40163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:17:51.291090 1477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:17:51.292050 1477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260618 22:17:51.300153 1477 server_base.cc:1034] running on GCE node
W20260618 22:17:51.300100 1482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:17:51.300119 1483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:17:51.300092 1485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:17:51.300915 1477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:17:51.301998 1477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:17:51.303175 1477 hybrid_clock.cc:648] HybridClock initialized: now 1781821071303157 us; error 30 us; skew 500 ppm
Jun 18 22:17:51 dist-test-slave-g6r4 krb5kdc[1464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821071, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:17:51.310637 1477 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260618 22:17:51.311501 1477 webserver.cc:460] Webserver started at http://127.0.98.126:36847/ using document root <none> and password file <none>
I20260618 22:17:51.311993 1477 fs_manager.cc:359] Metadata directory not provided
I20260618 22:17:51.312057 1477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:17:51.312224 1477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:17:51.314232 1477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/minicluster-data/master-0/data/instance:
uuid: "36253638f49349559f799f12de5095b6"
format_stamp: "Formatted at 2026-06-18 22:17:51 on dist-test-slave-g6r4"
server_key: "78003acebb74c055855040d8dbd87ffb"
server_key_iv: "94212f1582c49c06edde9f3755491b5c"
server_key_version: "encryptionkey@0"
I20260618 22:17:51.314735 1477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/minicluster-data/master-0/wal/instance:
uuid: "36253638f49349559f799f12de5095b6"
format_stamp: "Formatted at 2026-06-18 22:17:51 on dist-test-slave-g6r4"
server_key: "78003acebb74c055855040d8dbd87ffb"
server_key_iv: "94212f1582c49c06edde9f3755491b5c"
server_key_version: "encryptionkey@0"
I20260618 22:17:51.318218 1477 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260618 22:17:51.320359 1492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:17:51.321497 1477 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:17:51.321622 1477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/minicluster-data/master-0/wal
uuid: "36253638f49349559f799f12de5095b6"
format_stamp: "Formatted at 2026-06-18 22:17:51 on dist-test-slave-g6r4"
server_key: "78003acebb74c055855040d8dbd87ffb"
server_key_iv: "94212f1582c49c06edde9f3755491b5c"
server_key_version: "encryptionkey@0"
I20260618 22:17:51.321723 1477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:17:51.345093 1477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:17:51.348691 1477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:17:51.348944 1477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:17:51.357311 1477 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:40163
I20260618 22:17:51.357317 1554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:40163 every 8 connection(s)
I20260618 22:17:51.358307 1477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
I20260618 22:17:51.361027 1555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:17:51.362119 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 1477
I20260618 22:17:51.362241 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/minicluster-data/master-0/wal/instance
I20260618 22:17:51.362527 393 external_mini_cluster.cc:1428] Setting key 522a10e4915eea7faf7a6af2f1f255d1
I20260618 22:17:51.367103 1555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36253638f49349559f799f12de5095b6: Bootstrap starting.
I20260618 22:17:51.369249 1555 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 36253638f49349559f799f12de5095b6: Neither blocks nor log segments found. Creating new log.
I20260618 22:17:51.369987 1555 log.cc:824] T 00000000000000000000000000000000 P 36253638f49349559f799f12de5095b6: Log is configured to *not* fsync() on all Append() calls
Jun 18 22:17:51 dist-test-slave-g6r4 krb5kdc[1464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:17:51.372146 1555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 36253638f49349559f799f12de5095b6: No bootstrap required, opened a new log
I20260618 22:17:51.375083 1555 raft_consensus.cc:348] T 00000000000000000000000000000000 P 36253638f49349559f799f12de5095b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36253638f49349559f799f12de5095b6" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40163 } }
I20260618 22:17:51.375386 1555 raft_consensus.cc:374] T 00000000000000000000000000000000 P 36253638f49349559f799f12de5095b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:17:51.375471 1555 raft_consensus.cc:729] T 00000000000000000000000000000000 P 36253638f49349559f799f12de5095b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36253638f49349559f799f12de5095b6, State: Initialized, Role: FOLLOWER
I20260618 22:17:51.375924 1555 consensus_queue.cc:260] T 00000000000000000000000000000000 P 36253638f49349559f799f12de5095b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36253638f49349559f799f12de5095b6" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40163 } }
I20260618 22:17:51.376082 1555 raft_consensus.cc:388] T 00000000000000000000000000000000 P 36253638f49349559f799f12de5095b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:17:51.376153 1555 raft_consensus.cc:482] T 00000000000000000000000000000000 P 36253638f49349559f799f12de5095b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:17:51.376258 1555 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 36253638f49349559f799f12de5095b6 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:17:51.377136 1555 raft_consensus.cc:504] T 00000000000000000000000000000000 P 36253638f49349559f799f12de5095b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36253638f49349559f799f12de5095b6" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40163 } }
I20260618 22:17:51.377410 1555 leader_election.cc:302] T 00000000000000000000000000000000 P 36253638f49349559f799f12de5095b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 36253638f49349559f799f12de5095b6; no voters:
I20260618 22:17:51.377458 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:51.363980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56074 (local address 127.0.98.126:40163)
0618 22:17:51.364379 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0618 22:17:51.364393 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0618 22:17:51.364426 (+ 33us) server_negotiation.cc:407] Connection header received
0618 22:17:51.365072 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:17:51.365090 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:17:51.365418 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:17:51.365758 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:17:51.366619 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:51.367642 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:51.368462 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:51.368917 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:51.369089 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:51.371930 (+ 2841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:17:51.371935 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:17:51.371949 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:17:51.371974 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:17:51.374502 (+ 2528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:17:51.374965 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:17:51.374970 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:17:51.374975 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:17:51.375082 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:17:51.375423 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:17:51.375426 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:17:51.375433 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:17:51.375901 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:17:51.376089 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0618 22:17:51.376407 (+ 318us) server_negotiation.cc:299] Negotiation successful
0618 22:17:51.376704 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":117,"threads_started":1}
I20260618 22:17:51.377696 1555 leader_election.cc:288] T 00000000000000000000000000000000 P 36253638f49349559f799f12de5095b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:17:51.377867 1560 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 36253638f49349559f799f12de5095b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:17:51.378100 1560 raft_consensus.cc:686] T 00000000000000000000000000000000 P 36253638f49349559f799f12de5095b6 [term 1 LEADER]: Becoming Leader. State: Replica: 36253638f49349559f799f12de5095b6, State: Running, Role: LEADER
I20260618 22:17:51.378417 1560 consensus_queue.cc:237] T 00000000000000000000000000000000 P 36253638f49349559f799f12de5095b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36253638f49349559f799f12de5095b6" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40163 } }
I20260618 22:17:51.378906 1555 sys_catalog.cc:564] T 00000000000000000000000000000000 P 36253638f49349559f799f12de5095b6 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:17:51.380105 1561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36253638f49349559f799f12de5095b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "36253638f49349559f799f12de5095b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36253638f49349559f799f12de5095b6" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40163 } } }
I20260618 22:17:51.380095 1562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 36253638f49349559f799f12de5095b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 36253638f49349559f799f12de5095b6. Latest consensus state: current_term: 1 leader_uuid: "36253638f49349559f799f12de5095b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "36253638f49349559f799f12de5095b6" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40163 } } }
I20260618 22:17:51.380229 1561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36253638f49349559f799f12de5095b6 [sys.catalog]: This master's current role is: LEADER
I20260618 22:17:51.380229 1562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 36253638f49349559f799f12de5095b6 [sys.catalog]: This master's current role is: LEADER
I20260618 22:17:51.380721 1570 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:17:51.383664 1570 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:17:51.388888 1570 catalog_manager.cc:1347] Generated new cluster ID: 80c01aecb39e42efb31b2230529a7b6d
I20260618 22:17:51.388954 1570 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:17:51.394951 1570 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:17:51.396072 1570 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:17:51.404790 1570 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 36253638f49349559f799f12de5095b6: Generated new TSK 0
I20260618 22:17:51.405421 1570 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab.
I20260618 22:17:51.437135 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:40163
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:38803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:17:51.544741 1581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:17:51.545004 1581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:17:51.545079 1581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:17:51.548974 1581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:17:51.549050 1581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:17:51.549109 1581 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260618 22:17:51.549156 1581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260618 22:17:51.554186 1581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:38803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.126:40163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:17:51.555215 1581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:17:51.556226 1581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:17:51.565462 1586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:17:51.565512 1581 server_base.cc:1034] running on GCE node
W20260618 22:17:51.565469 1587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:17:51.565476 1589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:17:51.566041 1581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:17:51.566627 1581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:17:51.567802 1581 hybrid_clock.cc:648] HybridClock initialized: now 1781821071567787 us; error 28 us; skew 500 ppm
Jun 18 22:17:51 dist-test-slave-g6r4 krb5kdc[1464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821071, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:17:51.570407 1581 init.cc:377] Logged in from keytab as kudu/127.0.98.65@KRBTEST.COM (short username kudu)
I20260618 22:17:51.571179 1581 webserver.cc:460] Webserver started at http://127.0.98.65:37293/ using document root <none> and password file <none>
I20260618 22:17:51.571702 1581 fs_manager.cc:359] Metadata directory not provided
I20260618 22:17:51.571791 1581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:17:51.572010 1581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:17:51.573972 1581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/minicluster-data/ts-0/data/instance:
uuid: "92e3dccb82484cc18b97acc131cd87e7"
format_stamp: "Formatted at 2026-06-18 22:17:51 on dist-test-slave-g6r4"
server_key: "9f6531954f04e3a0c1fb618c5e97f908"
server_key_iv: "0e84496c8f0f1053e4783e66cee3b39b"
server_key_version: "encryptionkey@0"
I20260618 22:17:51.574535 1581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/minicluster-data/ts-0/wal/instance:
uuid: "92e3dccb82484cc18b97acc131cd87e7"
format_stamp: "Formatted at 2026-06-18 22:17:51 on dist-test-slave-g6r4"
server_key: "9f6531954f04e3a0c1fb618c5e97f908"
server_key_iv: "0e84496c8f0f1053e4783e66cee3b39b"
server_key_version: "encryptionkey@0"
I20260618 22:17:51.577955 1581 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:17:51.580340 1596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:17:51.581598 1581 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:17:51.581722 1581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/minicluster-data/ts-0/wal
uuid: "92e3dccb82484cc18b97acc131cd87e7"
format_stamp: "Formatted at 2026-06-18 22:17:51 on dist-test-slave-g6r4"
server_key: "9f6531954f04e3a0c1fb618c5e97f908"
server_key_iv: "0e84496c8f0f1053e4783e66cee3b39b"
server_key_version: "encryptionkey@0"
I20260618 22:17:51.581822 1581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:17:51.593483 1581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:17:51.596240 1581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:17:51.596431 1581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:17:51.597733 1581 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:17:51.597803 1581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:17:51.597859 1581 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:17:51.597944 1581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 18 22:17:51 dist-test-slave-g6r4 krb5kdc[1464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821071, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:17:51.609139 1581 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:33939
I20260618 22:17:51.609184 1711 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:33939 every 8 connection(s)
I20260618 22:17:51.610185 1581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
I20260618 22:17:51.611636 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:51.598529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:49031 (local address 127.0.98.126:40163)
0618 22:17:51.598647 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0618 22:17:51.598650 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:17:51.599662 (+ 1012us) server_negotiation.cc:407] Connection header received
0618 22:17:51.600699 (+ 1037us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:17:51.600702 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:17:51.600756 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:17:51.600814 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:17:51.602523 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:51.603274 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:51.604586 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:51.604834 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:51.604960 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:51.607894 (+ 2934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:17:51.607898 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:17:51.607901 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:17:51.607925 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:17:51.609424 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:17:51.609995 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:17:51.609998 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:17:51.610000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:17:51.610077 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:17:51.610510 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:17:51.610513 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:17:51.610514 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:17:51.610792 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:17:51.610901 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0618 22:17:51.611421 (+ 520us) server_negotiation.cc:299] Negotiation successful
0618 22:17:51.611522 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260618 22:17:51.612398 1605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:51.599067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:40163 (local address 127.0.98.65:49031)
0618 22:17:51.599525 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0618 22:17:51.599563 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0618 22:17:51.600446 (+ 883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:17:51.601219 (+ 773us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:17:51.601231 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:17:51.601722 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:17:51.602338 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:51.602360 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:51.603381 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:51.603384 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:51.604471 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:51.604480 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:51.605009 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:51.605014 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:51.605253 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:51.605924 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:17:51.605948 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:17:51.607740 (+ 1792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:17:51.609524 (+ 1784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:17:51.609529 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:17:51.609540 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:17:51.609871 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:17:51.610167 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:17:51.610170 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:17:51.610172 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:17:51.610366 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:17:51.610953 (+ 587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:17:51.610958 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:17:51.611210 (+ 252us) client_negotiation.cc:769] Sending connection context
0618 22:17:51.611358 (+ 148us) client_negotiation.cc:240] Negotiation successful
0618 22:17:51.611630 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":135,"threads_started":1}
I20260618 22:17:51.613330 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 1581
I20260618 22:17:51.613443 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/minicluster-data/ts-0/wal/instance
I20260618 22:17:51.613744 393 external_mini_cluster.cc:1428] Setting key b54f1bbf652ec98aebd14ba674bdd322
I20260618 22:17:51.613929 1712 heartbeater.cc:347] Connected to a master server at 127.0.98.126:40163
I20260618 22:17:51.614099 1712 heartbeater.cc:464] Registering TS with master...
I20260618 22:17:51.614645 1712 heartbeater.cc:511] Master 127.0.98.126:40163 requested a full tablet report, sending...
I20260618 22:17:51.616142 1508 ts_manager.cc:194] Registered new tserver with Master: 92e3dccb82484cc18b97acc131cd87e7 (127.0.98.65:33939)
I20260618 22:17:51.617667 1508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.65@KRBTEST.COM'} at 127.0.98.65:49031
WARNING: no policy specified for kudu/127.0.98.66@KRBTEST.COM; defaulting to no policy
I20260618 22:17:51.624718 1719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:51.617177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:40163 (local address 127.0.98.65:41315)
0618 22:17:51.617448 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0618 22:17:51.617461 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:17:51.617554 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:17:51.617817 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:17:51.617819 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:17:51.618040 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:17:51.618166 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:51.618173 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:51.619259 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:51.619262 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:51.620106 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:51.620119 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:51.620594 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:51.620601 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:51.620729 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:51.621203 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:17:51.621222 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:17:51.621668 (+ 446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:17:51.623311 (+ 1643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:17:51.623314 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:17:51.623317 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:17:51.623564 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:17:51.623845 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:17:51.623847 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:17:51.623849 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:17:51.623941 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:17:51.624232 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:17:51.624234 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:17:51.624301 (+ 67us) client_negotiation.cc:769] Sending connection context
0618 22:17:51.624393 (+ 92us) client_negotiation.cc:240] Negotiation successful
0618 22:17:51.624489 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":85,"threads_started":1}
I20260618 22:17:51.624923 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:51.617269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:41315 (local address 127.0.98.126:40163)
0618 22:17:51.617371 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0618 22:17:51.617374 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:17:51.617533 (+ 159us) server_negotiation.cc:407] Connection header received
0618 22:17:51.617647 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:17:51.617652 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:17:51.617700 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:17:51.617787 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:17:51.618288 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:51.619121 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:51.620224 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:51.620473 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:51.620555 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:51.621799 (+ 1244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:17:51.621802 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:17:51.621805 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:17:51.621824 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:17:51.623199 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:17:51.623681 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:17:51.623684 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:17:51.623685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:17:51.623756 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:17:51.624015 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:17:51.624017 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:17:51.624018 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:17:51.624159 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:17:51.624240 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0618 22:17:51.624605 (+ 365us) server_negotiation.cc:299] Negotiation successful
0618 22:17:51.624749 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab.
I20260618 22:17:51.640748 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:40163
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:38803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:17:51.746462 1721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:17:51.746702 1721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:17:51.746744 1721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:17:51.750744 1721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:17:51.750793 1721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:17:51.750831 1721 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260618 22:17:51.750862 1721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260618 22:17:51.755787 1721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:38803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.126:40163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:17:51.756664 1721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:17:51.757565 1721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:17:51.766686 1727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:17:51.766681 1729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:17:51.766661 1726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:17:51.766992 1721 server_base.cc:1034] running on GCE node
I20260618 22:17:51.767364 1721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:17:51.767947 1721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:17:51.769124 1721 hybrid_clock.cc:648] HybridClock initialized: now 1781821071769113 us; error 30 us; skew 500 ppm
Jun 18 22:17:51 dist-test-slave-g6r4 krb5kdc[1464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821071, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:17:51.771682 1721 init.cc:377] Logged in from keytab as kudu/127.0.98.66@KRBTEST.COM (short username kudu)
I20260618 22:17:51.772462 1721 webserver.cc:460] Webserver started at http://127.0.98.66:34005/ using document root <none> and password file <none>
I20260618 22:17:51.772944 1721 fs_manager.cc:359] Metadata directory not provided
I20260618 22:17:51.773001 1721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:17:51.773151 1721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:17:51.775053 1721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/minicluster-data/ts-1/data/instance:
uuid: "0ec8274e1c6846f2b89e28a0856b60e3"
format_stamp: "Formatted at 2026-06-18 22:17:51 on dist-test-slave-g6r4"
server_key: "1ade178f8a6d9fceb3c345abeb9fe207"
server_key_iv: "6ad236d7105113dd777cb504009bc5f0"
server_key_version: "encryptionkey@0"
I20260618 22:17:51.775547 1721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/minicluster-data/ts-1/wal/instance:
uuid: "0ec8274e1c6846f2b89e28a0856b60e3"
format_stamp: "Formatted at 2026-06-18 22:17:51 on dist-test-slave-g6r4"
server_key: "1ade178f8a6d9fceb3c345abeb9fe207"
server_key_iv: "6ad236d7105113dd777cb504009bc5f0"
server_key_version: "encryptionkey@0"
I20260618 22:17:51.778978 1721 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260618 22:17:51.781587 1736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:17:51.782802 1721 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:17:51.782936 1721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/minicluster-data/ts-1/wal
uuid: "0ec8274e1c6846f2b89e28a0856b60e3"
format_stamp: "Formatted at 2026-06-18 22:17:51 on dist-test-slave-g6r4"
server_key: "1ade178f8a6d9fceb3c345abeb9fe207"
server_key_iv: "6ad236d7105113dd777cb504009bc5f0"
server_key_version: "encryptionkey@0"
I20260618 22:17:51.783037 1721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:17:51.840126 1721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:17:51.843168 1721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:17:51.843367 1721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:17:51.844720 1721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:17:51.844792 1721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:17:51.844856 1721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:17:51.844900 1721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 18 22:17:51 dist-test-slave-g6r4 krb5kdc[1464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821071, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:17:51.855934 1721 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:39177
I20260618 22:17:51.855952 1851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:39177 every 8 connection(s)
I20260618 22:17:51.856812 1721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
I20260618 22:17:51.857180 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 1721
I20260618 22:17:51.857302 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/minicluster-data/ts-1/wal/instance
I20260618 22:17:51.857589 393 external_mini_cluster.cc:1428] Setting key 30f43da5a047b5e499e96f81c1b5c82d
I20260618 22:17:51.859256 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:51.845607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:48351 (local address 127.0.98.126:40163)
0618 22:17:51.845732 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0618 22:17:51.845735 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:17:51.846963 (+ 1228us) server_negotiation.cc:407] Connection header received
0618 22:17:51.847977 (+ 1014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:17:51.847983 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:17:51.848045 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:17:51.848104 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:17:51.850294 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:51.851190 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:51.852607 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:51.852855 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:51.852972 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:51.855472 (+ 2500us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:17:51.855475 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:17:51.855477 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:17:51.855499 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:17:51.857045 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:17:51.857629 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:17:51.857633 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:17:51.857635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:17:51.857713 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:17:51.858147 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:17:51.858153 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:17:51.858158 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:17:51.858367 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:17:51.858449 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0618 22:17:51.858976 (+ 527us) server_negotiation.cc:299] Negotiation successful
0618 22:17:51.859093 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260618 22:17:51.859963 1745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:51.846127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:40163 (local address 127.0.98.66:48351)
0618 22:17:51.846767 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0618 22:17:51.846850 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0618 22:17:51.847734 (+ 884us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:17:51.848502 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:17:51.848512 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:17:51.849014 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:17:51.850112 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:51.850134 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:51.851343 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:51.851346 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:51.852481 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:51.852489 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:51.853007 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:51.853011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:51.853183 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:51.853740 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:17:51.853755 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:17:51.855303 (+ 1548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:17:51.857146 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:17:51.857152 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:17:51.857166 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:17:51.857468 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:17:51.857803 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:17:51.857807 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:17:51.857810 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:17:51.858010 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:17:51.858466 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:17:51.858472 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:17:51.858728 (+ 256us) client_negotiation.cc:769] Sending connection context
0618 22:17:51.858917 (+ 189us) client_negotiation.cc:240] Negotiation successful
0618 22:17:51.859178 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":143,"threads_started":1}
I20260618 22:17:51.861634 1852 heartbeater.cc:347] Connected to a master server at 127.0.98.126:40163
I20260618 22:17:51.861765 1852 heartbeater.cc:464] Registering TS with master...
I20260618 22:17:51.862318 1852 heartbeater.cc:511] Master 127.0.98.126:40163 requested a full tablet report, sending...
I20260618 22:17:51.863359 1509 ts_manager.cc:194] Registered new tserver with Master: 0ec8274e1c6846f2b89e28a0856b60e3 (127.0.98.66:39177)
I20260618 22:17:51.864243 1509 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.66@KRBTEST.COM'} at 127.0.98.66:48351
WARNING: no policy specified for kudu/127.0.98.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.67@KRBTEST.COM" created.
I20260618 22:17:51.874513 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:51.867243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:33155 (local address 127.0.98.126:40163)
0618 22:17:51.867367 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0618 22:17:51.867371 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:17:51.867526 (+ 155us) server_negotiation.cc:407] Connection header received
0618 22:17:51.867662 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:17:51.867665 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:17:51.867746 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:17:51.867825 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:17:51.868238 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:51.868821 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:51.869739 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:51.870035 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:51.870139 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:51.871362 (+ 1223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:17:51.871365 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:17:51.871367 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:17:51.871387 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:17:51.872793 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:17:51.873246 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:17:51.873248 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:17:51.873249 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:17:51.873316 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:17:51.873658 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:17:51.873661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:17:51.873662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:17:51.873860 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:17:51.874026 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0618 22:17:51.874323 (+ 297us) server_negotiation.cc:299] Negotiation successful
0618 22:17:51.874410 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260618 22:17:51.874776 1859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:51.867186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:40163 (local address 127.0.98.66:33155)
0618 22:17:51.867447 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0618 22:17:51.867460 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:17:51.867569 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:17:51.867842 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:17:51.867845 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:17:51.868031 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:17:51.868122 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:51.868127 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:51.868943 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:51.868947 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:51.869627 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:51.869636 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:51.870141 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:51.870145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:51.870236 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:51.870780 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:17:51.870798 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:17:51.871238 (+ 440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:17:51.872893 (+ 1655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:17:51.872897 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:17:51.872899 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:17:51.873153 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:17:51.873404 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:17:51.873410 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:17:51.873414 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:17:51.873520 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:17:51.874028 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:17:51.874034 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:17:51.874149 (+ 115us) client_negotiation.cc:769] Sending connection context
0618 22:17:51.874281 (+ 132us) client_negotiation.cc:240] Negotiation successful
0618 22:17:51.874444 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":77,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab.
I20260618 22:17:51.885490 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:40163
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:38803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:17:51.992846 1861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:17:51.993089 1861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:17:51.993131 1861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:17:51.997026 1861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:17:51.997087 1861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:17:51.997123 1861 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260618 22:17:51.997154 1861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260618 22:17:52.002125 1861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:38803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.126:40163
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1861
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:17:52.003026 1861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:17:52.003916 1861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:17:52.012717 1866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:17:52.012794 1861 server_base.cc:1034] running on GCE node
W20260618 22:17:52.012723 1867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:17:52.012735 1869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:17:52.013331 1861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:17:52.013999 1861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:17:52.015163 1861 hybrid_clock.cc:648] HybridClock initialized: now 1781821072015154 us; error 31 us; skew 500 ppm
Jun 18 22:17:52 dist-test-slave-g6r4 krb5kdc[1464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821072, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:17:52.017773 1861 init.cc:377] Logged in from keytab as kudu/127.0.98.67@KRBTEST.COM (short username kudu)
I20260618 22:17:52.018570 1861 webserver.cc:460] Webserver started at http://127.0.98.67:40581/ using document root <none> and password file <none>
I20260618 22:17:52.019124 1861 fs_manager.cc:359] Metadata directory not provided
I20260618 22:17:52.019217 1861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:17:52.019420 1861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:17:52.021358 1861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/minicluster-data/ts-2/data/instance:
uuid: "71d4b767e7a841c584be081a7f4a3dbb"
format_stamp: "Formatted at 2026-06-18 22:17:52 on dist-test-slave-g6r4"
server_key: "ae21cfb830053cb3d09ec70df5bce877"
server_key_iv: "d2af2706ccd14e76e52835aa878f3e51"
server_key_version: "encryptionkey@0"
I20260618 22:17:52.021883 1861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/minicluster-data/ts-2/wal/instance:
uuid: "71d4b767e7a841c584be081a7f4a3dbb"
format_stamp: "Formatted at 2026-06-18 22:17:52 on dist-test-slave-g6r4"
server_key: "ae21cfb830053cb3d09ec70df5bce877"
server_key_iv: "d2af2706ccd14e76e52835aa878f3e51"
server_key_version: "encryptionkey@0"
I20260618 22:17:52.025331 1861 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260618 22:17:52.027715 1876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:17:52.028844 1861 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:17:52.028981 1861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/minicluster-data/ts-2/wal
uuid: "71d4b767e7a841c584be081a7f4a3dbb"
format_stamp: "Formatted at 2026-06-18 22:17:52 on dist-test-slave-g6r4"
server_key: "ae21cfb830053cb3d09ec70df5bce877"
server_key_iv: "d2af2706ccd14e76e52835aa878f3e51"
server_key_version: "encryptionkey@0"
I20260618 22:17:52.029083 1861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:17:52.045806 1861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:17:52.048833 1861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:17:52.049021 1861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:17:52.050427 1861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:17:52.050490 1861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:17:52.050546 1861 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:17:52.050578 1861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 18 22:17:52 dist-test-slave-g6r4 krb5kdc[1464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821072, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:17:52.062057 1861 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:34949
I20260618 22:17:52.062103 1991 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:34949 every 8 connection(s)
I20260618 22:17:52.063043 1861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
I20260618 22:17:52.065248 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:52.051085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:37967 (local address 127.0.98.126:40163)
0618 22:17:52.051222 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0618 22:17:52.051227 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:17:52.052095 (+ 868us) server_negotiation.cc:407] Connection header received
0618 22:17:52.053386 (+ 1291us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:17:52.053391 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:17:52.053455 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:17:52.053685 (+ 230us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:17:52.055554 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:52.056372 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:52.058243 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:52.058608 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:52.058720 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:52.061471 (+ 2751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:17:52.061476 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:17:52.061479 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:17:52.061507 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:17:52.063251 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:17:52.063799 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:17:52.063805 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:17:52.063807 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:17:52.063887 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:17:52.064205 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:17:52.064210 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:17:52.064212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:17:52.064393 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:17:52.064496 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0618 22:17:52.064938 (+ 442us) server_negotiation.cc:299] Negotiation successful
0618 22:17:52.065049 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260618 22:17:52.065973 1885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:52.051460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:40163 (local address 127.0.98.67:37967)
0618 22:17:52.051922 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0618 22:17:52.051975 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0618 22:17:52.053128 (+ 1153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:17:52.054186 (+ 1058us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:17:52.054196 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:17:52.054658 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:17:52.055337 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:52.055386 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:52.056513 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:52.056521 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:52.058074 (+ 1553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:52.058087 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:52.058767 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:52.058772 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:52.058973 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:52.059548 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:17:52.059574 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:17:52.061315 (+ 1741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:17:52.063354 (+ 2039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:17:52.063360 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:17:52.063373 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:17:52.063699 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:17:52.063981 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:17:52.063984 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:17:52.063986 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:17:52.064115 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:17:52.064492 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:17:52.064498 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:17:52.064716 (+ 218us) client_negotiation.cc:769] Sending connection context
0618 22:17:52.064897 (+ 181us) client_negotiation.cc:240] Negotiation successful
0618 22:17:52.065216 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":98,"threads_started":1}
I20260618 22:17:52.067131 1992 heartbeater.cc:347] Connected to a master server at 127.0.98.126:40163
I20260618 22:17:52.067281 1992 heartbeater.cc:464] Registering TS with master...
I20260618 22:17:52.067770 1992 heartbeater.cc:511] Master 127.0.98.126:40163 requested a full tablet report, sending...
I20260618 22:17:52.068774 1508 ts_manager.cc:194] Registered new tserver with Master: 71d4b767e7a841c584be081a7f4a3dbb (127.0.98.67:34949)
I20260618 22:17:52.069619 1508 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.67@KRBTEST.COM'} at 127.0.98.67:37967
I20260618 22:17:52.071347 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 1861
I20260618 22:17:52.071462 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/minicluster-data/ts-2/wal/instance
I20260618 22:17:52.071759 393 external_mini_cluster.cc:1428] Setting key 840be5921a2f1699fab4ed27df96c25d
I20260618 22:17:52.073633 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 22:17:52.075767 1998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:52.068957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:40163 (local address 127.0.98.67:35455)
0618 22:17:52.069228 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0618 22:17:52.069243 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:17:52.069336 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:17:52.069539 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:17:52.069542 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:17:52.069730 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:17:52.069836 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:52.069844 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:52.070576 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:52.070579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:52.071164 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:52.071174 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:52.071682 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:52.071686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:52.071799 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:52.072335 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:17:52.072348 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:17:52.072672 (+ 324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:17:52.074369 (+ 1697us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:17:52.074373 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:17:52.074375 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:17:52.074560 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:17:52.074842 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:17:52.074845 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:17:52.074847 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:17:52.074926 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:17:52.075262 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:17:52.075265 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:17:52.075350 (+ 85us) client_negotiation.cc:769] Sending connection context
0618 22:17:52.075455 (+ 105us) client_negotiation.cc:240] Negotiation successful
0618 22:17:52.075588 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":106,"threads_started":1}
I20260618 22:17:52.075767 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:52.069003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:35455 (local address 127.0.98.126:40163)
0618 22:17:52.069120 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0618 22:17:52.069124 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:17:52.069311 (+ 187us) server_negotiation.cc:407] Connection header received
0618 22:17:52.069408 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:17:52.069410 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:17:52.069443 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:17:52.069504 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:17:52.069968 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:52.070455 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:52.071290 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:52.071563 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:52.071656 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:52.072796 (+ 1140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:17:52.072800 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:17:52.072802 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:17:52.072826 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:17:52.074249 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:17:52.074657 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:17:52.074662 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:17:52.074664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:17:52.074736 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:17:52.075010 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:17:52.075015 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:17:52.075017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:17:52.075172 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:17:52.075272 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0618 22:17:52.075505 (+ 233us) server_negotiation.cc:299] Negotiation successful
0618 22:17:52.075612 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":48}
Jun 18 22:17:52 dist-test-slave-g6r4 krb5kdc[1464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821072, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 18 22:17:52 dist-test-slave-g6r4 krb5kdc[1464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821072, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:17:52.098368 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:52.090785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56076 (local address 127.0.98.126:40163)
0618 22:17:52.090892 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0618 22:17:52.090895 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:17:52.090906 (+ 11us) server_negotiation.cc:407] Connection header received
0618 22:17:52.090937 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:17:52.090939 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:17:52.090970 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:17:52.091038 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:17:52.091706 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:52.092193 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:52.093000 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:52.093192 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:52.093264 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:52.095604 (+ 2340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:17:52.095606 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:17:52.095608 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:17:52.095628 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:17:52.096872 (+ 1244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:17:52.097275 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:17:52.097281 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:17:52.097285 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:17:52.097363 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:17:52.097671 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:17:52.097673 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:17:52.097674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:17:52.097832 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:17:52.097989 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0618 22:17:52.098182 (+ 193us) server_negotiation.cc:299] Negotiation successful
0618 22:17:52.098266 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260618 22:17:52.100980 1508 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56076:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 22:17:52.103998 1508 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:17:52.117185 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:52.111860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:39177 (local address 127.0.0.1:60282)
0618 22:17:52.112649 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0618 22:17:52.112662 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:17:52.112753 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:17:52.113239 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:17:52.113246 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:17:52.113266 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:17:52.113410 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:52.113417 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:52.114516 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:52.114520 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:52.115789 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:52.115798 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:52.116639 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:52.116642 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:52.116782 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:52.116826 (+ 44us) client_negotiation.cc:769] Sending connection context
0618 22:17:52.116939 (+ 113us) client_negotiation.cc:240] Negotiation successful
0618 22:17:52.117050 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":728,"mutex_wait_us":1,"thread_start_us":85,"threads_started":1}
I20260618 22:17:52.117197 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:52.112241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60282 (local address 127.0.98.66:39177)
0618 22:17:52.112629 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0618 22:17:52.112643 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0618 22:17:52.112782 (+ 139us) server_negotiation.cc:407] Connection header received
0618 22:17:52.112875 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:17:52.112880 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:17:52.113083 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:17:52.113239 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:17:52.113519 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:52.114414 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:52.115930 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:52.116521 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:52.116641 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:52.116793 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0618 22:17:52.116977 (+ 184us) server_negotiation.cc:299] Negotiation successful
0618 22:17:52.117058 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":104,"threads_started":1}
I20260618 22:17:52.117640 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:52.111395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:34949 (local address 127.0.0.1:51628)
0618 22:17:52.111710 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0618 22:17:52.111758 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0618 22:17:52.111878 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:17:52.113396 (+ 1518us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:17:52.113401 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:17:52.113417 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:17:52.113548 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:52.113557 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:52.114623 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:52.114629 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:52.115993 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:52.116002 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:52.117005 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:52.117008 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:52.117355 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:52.117377 (+ 22us) client_negotiation.cc:769] Sending connection context
0618 22:17:52.117447 (+ 70us) client_negotiation.cc:240] Negotiation successful
0618 22:17:52.117512 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":235,"thread_start_us":74,"threads_started":1}
I20260618 22:17:52.118191 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:52.112094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51628 (local address 127.0.98.67:34949)
0618 22:17:52.112974 (+ 880us) server_negotiation.cc:206] Beginning negotiation
0618 22:17:52.112982 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0618 22:17:52.112999 (+ 17us) server_negotiation.cc:407] Connection header received
0618 22:17:52.113053 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:17:52.113056 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:17:52.113222 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:17:52.113413 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:17:52.113702 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:52.114516 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:52.116104 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:52.116874 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:52.117077 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:52.117246 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0618 22:17:52.117938 (+ 692us) server_negotiation.cc:299] Negotiation successful
0618 22:17:52.118030 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":797,"thread_start_us":115,"threads_started":1}
I20260618 22:17:52.118558 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:52.111543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:33939 (local address 127.0.0.1:45820)
0618 22:17:52.112468 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0618 22:17:52.112482 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:17:52.112564 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:17:52.113413 (+ 849us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:17:52.113415 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:17:52.113426 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:17:52.114555 (+ 1129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:52.114561 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:52.115897 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:52.115901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:52.117059 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:52.117067 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:52.118145 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:52.118148 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:52.118255 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:52.118278 (+ 23us) client_negotiation.cc:769] Sending connection context
0618 22:17:52.118368 (+ 90us) client_negotiation.cc:240] Negotiation successful
0618 22:17:52.118439 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":866,"mutex_wait_us":1083,"thread_start_us":94,"threads_started":1}
I20260618 22:17:52.118628 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:52.112191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45820 (local address 127.0.98.65:33939)
0618 22:17:52.112926 (+ 735us) server_negotiation.cc:206] Beginning negotiation
0618 22:17:52.112939 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0618 22:17:52.112958 (+ 19us) server_negotiation.cc:407] Connection header received
0618 22:17:52.113016 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:17:52.113020 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:17:52.113177 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:17:52.113292 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:17:52.114700 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:52.115787 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:52.117178 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:52.118034 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:52.118136 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:52.118287 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0618 22:17:52.118393 (+ 106us) server_negotiation.cc:299] Negotiation successful
0618 22:17:52.118494 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":75,"threads_started":1}
I20260618 22:17:52.120458 1786 tablet_service.cc:1467] Processing CreateTablet for tablet 6d660c1b36c847a7994237b228122339 (DEFAULT_TABLE table=test-table [id=1daf54b54dca4924a8b57abf9c61fcae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:17:52.120854 1646 tablet_service.cc:1467] Processing CreateTablet for tablet 6d660c1b36c847a7994237b228122339 (DEFAULT_TABLE table=test-table [id=1daf54b54dca4924a8b57abf9c61fcae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:17:52.120841 1926 tablet_service.cc:1467] Processing CreateTablet for tablet 6d660c1b36c847a7994237b228122339 (DEFAULT_TABLE table=test-table [id=1daf54b54dca4924a8b57abf9c61fcae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:17:52.121510 1786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d660c1b36c847a7994237b228122339. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:17:52.121694 1926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d660c1b36c847a7994237b228122339. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:17:52.121752 1646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d660c1b36c847a7994237b228122339. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:17:52.127378 2015 tablet_bootstrap.cc:492] T 6d660c1b36c847a7994237b228122339 P 92e3dccb82484cc18b97acc131cd87e7: Bootstrap starting.
I20260618 22:17:52.129534 2017 tablet_bootstrap.cc:492] T 6d660c1b36c847a7994237b228122339 P 71d4b767e7a841c584be081a7f4a3dbb: Bootstrap starting.
I20260618 22:17:52.129568 2015 tablet_bootstrap.cc:654] T 6d660c1b36c847a7994237b228122339 P 92e3dccb82484cc18b97acc131cd87e7: Neither blocks nor log segments found. Creating new log.
I20260618 22:17:52.129644 2016 tablet_bootstrap.cc:492] T 6d660c1b36c847a7994237b228122339 P 0ec8274e1c6846f2b89e28a0856b60e3: Bootstrap starting.
I20260618 22:17:52.130637 2015 log.cc:824] T 6d660c1b36c847a7994237b228122339 P 92e3dccb82484cc18b97acc131cd87e7: Log is configured to *not* fsync() on all Append() calls
I20260618 22:17:52.131505 2016 tablet_bootstrap.cc:654] T 6d660c1b36c847a7994237b228122339 P 0ec8274e1c6846f2b89e28a0856b60e3: Neither blocks nor log segments found. Creating new log.
I20260618 22:17:52.131505 2017 tablet_bootstrap.cc:654] T 6d660c1b36c847a7994237b228122339 P 71d4b767e7a841c584be081a7f4a3dbb: Neither blocks nor log segments found. Creating new log.
I20260618 22:17:52.132221 2015 tablet_bootstrap.cc:492] T 6d660c1b36c847a7994237b228122339 P 92e3dccb82484cc18b97acc131cd87e7: No bootstrap required, opened a new log
I20260618 22:17:52.132293 2016 log.cc:824] T 6d660c1b36c847a7994237b228122339 P 0ec8274e1c6846f2b89e28a0856b60e3: Log is configured to *not* fsync() on all Append() calls
I20260618 22:17:52.132313 2017 log.cc:824] T 6d660c1b36c847a7994237b228122339 P 71d4b767e7a841c584be081a7f4a3dbb: Log is configured to *not* fsync() on all Append() calls
I20260618 22:17:52.132473 2015 ts_tablet_manager.cc:1397] T 6d660c1b36c847a7994237b228122339 P 92e3dccb82484cc18b97acc131cd87e7: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260618 22:17:52.133865 2016 tablet_bootstrap.cc:492] T 6d660c1b36c847a7994237b228122339 P 0ec8274e1c6846f2b89e28a0856b60e3: No bootstrap required, opened a new log
I20260618 22:17:52.134049 2016 ts_tablet_manager.cc:1397] T 6d660c1b36c847a7994237b228122339 P 0ec8274e1c6846f2b89e28a0856b60e3: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260618 22:17:52.135658 2015 raft_consensus.cc:348] T 6d660c1b36c847a7994237b228122339 P 92e3dccb82484cc18b97acc131cd87e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d4b767e7a841c584be081a7f4a3dbb" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34949 } } peers { permanent_uuid: "92e3dccb82484cc18b97acc131cd87e7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33939 } } peers { permanent_uuid: "0ec8274e1c6846f2b89e28a0856b60e3" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39177 } }
I20260618 22:17:52.135852 2015 raft_consensus.cc:374] T 6d660c1b36c847a7994237b228122339 P 92e3dccb82484cc18b97acc131cd87e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:17:52.135907 2015 raft_consensus.cc:729] T 6d660c1b36c847a7994237b228122339 P 92e3dccb82484cc18b97acc131cd87e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92e3dccb82484cc18b97acc131cd87e7, State: Initialized, Role: FOLLOWER
I20260618 22:17:52.136317 2015 consensus_queue.cc:260] T 6d660c1b36c847a7994237b228122339 P 92e3dccb82484cc18b97acc131cd87e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d4b767e7a841c584be081a7f4a3dbb" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34949 } } peers { permanent_uuid: "92e3dccb82484cc18b97acc131cd87e7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33939 } } peers { permanent_uuid: "0ec8274e1c6846f2b89e28a0856b60e3" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39177 } }
I20260618 22:17:52.136890 1712 heartbeater.cc:503] Master 127.0.98.126:40163 was elected leader, sending a full tablet report...
I20260618 22:17:52.136767 2016 raft_consensus.cc:348] T 6d660c1b36c847a7994237b228122339 P 0ec8274e1c6846f2b89e28a0856b60e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d4b767e7a841c584be081a7f4a3dbb" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34949 } } peers { permanent_uuid: "92e3dccb82484cc18b97acc131cd87e7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33939 } } peers { permanent_uuid: "0ec8274e1c6846f2b89e28a0856b60e3" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39177 } }
I20260618 22:17:52.136979 2016 raft_consensus.cc:374] T 6d660c1b36c847a7994237b228122339 P 0ec8274e1c6846f2b89e28a0856b60e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:17:52.137068 2016 raft_consensus.cc:729] T 6d660c1b36c847a7994237b228122339 P 0ec8274e1c6846f2b89e28a0856b60e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ec8274e1c6846f2b89e28a0856b60e3, State: Initialized, Role: FOLLOWER
I20260618 22:17:52.137205 2015 ts_tablet_manager.cc:1428] T 6d660c1b36c847a7994237b228122339 P 92e3dccb82484cc18b97acc131cd87e7: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260618 22:17:52.137463 2016 consensus_queue.cc:260] T 6d660c1b36c847a7994237b228122339 P 0ec8274e1c6846f2b89e28a0856b60e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d4b767e7a841c584be081a7f4a3dbb" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34949 } } peers { permanent_uuid: "92e3dccb82484cc18b97acc131cd87e7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33939 } } peers { permanent_uuid: "0ec8274e1c6846f2b89e28a0856b60e3" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39177 } }
I20260618 22:17:52.138010 1852 heartbeater.cc:503] Master 127.0.98.126:40163 was elected leader, sending a full tablet report...
I20260618 22:17:52.138325 2016 ts_tablet_manager.cc:1428] T 6d660c1b36c847a7994237b228122339 P 0ec8274e1c6846f2b89e28a0856b60e3: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
I20260618 22:17:52.139098 2017 tablet_bootstrap.cc:492] T 6d660c1b36c847a7994237b228122339 P 71d4b767e7a841c584be081a7f4a3dbb: No bootstrap required, opened a new log
I20260618 22:17:52.139266 2017 ts_tablet_manager.cc:1397] T 6d660c1b36c847a7994237b228122339 P 71d4b767e7a841c584be081a7f4a3dbb: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20260618 22:17:52.142637 2017 raft_consensus.cc:348] T 6d660c1b36c847a7994237b228122339 P 71d4b767e7a841c584be081a7f4a3dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d4b767e7a841c584be081a7f4a3dbb" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34949 } } peers { permanent_uuid: "92e3dccb82484cc18b97acc131cd87e7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33939 } } peers { permanent_uuid: "0ec8274e1c6846f2b89e28a0856b60e3" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39177 } }
I20260618 22:17:52.142813 2017 raft_consensus.cc:374] T 6d660c1b36c847a7994237b228122339 P 71d4b767e7a841c584be081a7f4a3dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:17:52.142891 2017 raft_consensus.cc:729] T 6d660c1b36c847a7994237b228122339 P 71d4b767e7a841c584be081a7f4a3dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71d4b767e7a841c584be081a7f4a3dbb, State: Initialized, Role: FOLLOWER
I20260618 22:17:52.143249 2017 consensus_queue.cc:260] T 6d660c1b36c847a7994237b228122339 P 71d4b767e7a841c584be081a7f4a3dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d4b767e7a841c584be081a7f4a3dbb" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34949 } } peers { permanent_uuid: "92e3dccb82484cc18b97acc131cd87e7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33939 } } peers { permanent_uuid: "0ec8274e1c6846f2b89e28a0856b60e3" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39177 } }
I20260618 22:17:52.143818 1992 heartbeater.cc:503] Master 127.0.98.126:40163 was elected leader, sending a full tablet report...
I20260618 22:17:52.143990 2017 ts_tablet_manager.cc:1428] T 6d660c1b36c847a7994237b228122339 P 71d4b767e7a841c584be081a7f4a3dbb: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260618 22:17:52.280058 2022 raft_consensus.cc:482] T 6d660c1b36c847a7994237b228122339 P 0ec8274e1c6846f2b89e28a0856b60e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:17:52.280259 2022 raft_consensus.cc:504] T 6d660c1b36c847a7994237b228122339 P 0ec8274e1c6846f2b89e28a0856b60e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d4b767e7a841c584be081a7f4a3dbb" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34949 } } peers { permanent_uuid: "92e3dccb82484cc18b97acc131cd87e7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33939 } } peers { permanent_uuid: "0ec8274e1c6846f2b89e28a0856b60e3" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39177 } }
I20260618 22:17:52.281522 2022 leader_election.cc:288] T 6d660c1b36c847a7994237b228122339 P 0ec8274e1c6846f2b89e28a0856b60e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71d4b767e7a841c584be081a7f4a3dbb (127.0.98.67:34949), 92e3dccb82484cc18b97acc131cd87e7 (127.0.98.65:33939)
I20260618 22:17:52.283751 2023 raft_consensus.cc:482] T 6d660c1b36c847a7994237b228122339 P 71d4b767e7a841c584be081a7f4a3dbb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:17:52.283932 2023 raft_consensus.cc:504] T 6d660c1b36c847a7994237b228122339 P 71d4b767e7a841c584be081a7f4a3dbb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d4b767e7a841c584be081a7f4a3dbb" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34949 } } peers { permanent_uuid: "92e3dccb82484cc18b97acc131cd87e7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33939 } } peers { permanent_uuid: "0ec8274e1c6846f2b89e28a0856b60e3" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39177 } }
I20260618 22:17:52.285189 2023 leader_election.cc:288] T 6d660c1b36c847a7994237b228122339 P 71d4b767e7a841c584be081a7f4a3dbb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92e3dccb82484cc18b97acc131cd87e7 (127.0.98.65:33939), 0ec8274e1c6846f2b89e28a0856b60e3 (127.0.98.66:39177)
I20260618 22:17:52.285425 1745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:52.281761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:33939 (local address 127.0.98.66:55401)
0618 22:17:52.281981 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0618 22:17:52.281997 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 22:17:52.282099 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:17:52.282396 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:17:52.282402 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:17:52.282420 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:17:52.282527 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:52.282531 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:52.283304 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:52.283310 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:52.284263 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:52.284276 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:52.284960 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:52.284963 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:52.285067 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:52.285097 (+ 30us) client_negotiation.cc:769] Sending connection context
0618 22:17:52.285191 (+ 94us) client_negotiation.cc:240] Negotiation successful
0618 22:17:52.285281 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":39}
I20260618 22:17:52.286175 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:52.281946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:55401 (local address 127.0.98.65:33939)
0618 22:17:52.282077 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0618 22:17:52.282086 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0618 22:17:52.282101 (+ 15us) server_negotiation.cc:407] Connection header received
0618 22:17:52.282209 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:17:52.282212 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:17:52.282279 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:17:52.282367 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:17:52.282670 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:52.283204 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:52.284366 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:52.284849 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:52.284960 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:52.285028 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0618 22:17:52.285950 (+ 922us) server_negotiation.cc:299] Negotiation successful
0618 22:17:52.286030 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260618 22:17:52.287053 1666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d660c1b36c847a7994237b228122339" candidate_uuid: "0ec8274e1c6846f2b89e28a0856b60e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92e3dccb82484cc18b97acc131cd87e7" is_pre_election: true
I20260618 22:17:52.287235 2024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:52.281782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:34949 (local address 127.0.98.66:45631)
0618 22:17:52.282172 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0618 22:17:52.282185 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:17:52.282270 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:17:52.282508 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:17:52.282511 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:17:52.282533 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:17:52.282649 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:52.282653 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:52.283481 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:52.283488 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:52.285843 (+ 2355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:52.285851 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:52.286850 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:52.286854 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:52.286943 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:52.286963 (+ 20us) client_negotiation.cc:769] Sending connection context
0618 22:17:52.287033 (+ 70us) client_negotiation.cc:240] Negotiation successful
0618 22:17:52.287096 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":129,"threads_started":1}
I20260618 22:17:52.287391 1666 raft_consensus.cc:2454] T 6d660c1b36c847a7994237b228122339 P 92e3dccb82484cc18b97acc131cd87e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ec8274e1c6846f2b89e28a0856b60e3 in term 0.
I20260618 22:17:52.287567 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:52.281854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:45631 (local address 127.0.98.67:34949)
0618 22:17:52.282000 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0618 22:17:52.282005 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:17:52.282270 (+ 265us) server_negotiation.cc:407] Connection header received
0618 22:17:52.282333 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:17:52.282339 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:17:52.282391 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:17:52.282461 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:17:52.282795 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:52.283377 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:52.285989 (+ 2612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:52.286599 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:52.286695 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:52.286778 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0618 22:17:52.287379 (+ 601us) server_negotiation.cc:299] Negotiation successful
0618 22:17:52.287467 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260618 22:17:52.287997 1931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d660c1b36c847a7994237b228122339" candidate_uuid: "0ec8274e1c6846f2b89e28a0856b60e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71d4b767e7a841c584be081a7f4a3dbb" is_pre_election: true
I20260618 22:17:52.288192 1931 raft_consensus.cc:2454] T 6d660c1b36c847a7994237b228122339 P 71d4b767e7a841c584be081a7f4a3dbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ec8274e1c6846f2b89e28a0856b60e3 in term 0.
I20260618 22:17:52.288273 1738 leader_election.cc:302] T 6d660c1b36c847a7994237b228122339 P 0ec8274e1c6846f2b89e28a0856b60e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ec8274e1c6846f2b89e28a0856b60e3, 92e3dccb82484cc18b97acc131cd87e7; no voters:
I20260618 22:17:52.288561 2022 raft_consensus.cc:2781] T 6d660c1b36c847a7994237b228122339 P 0ec8274e1c6846f2b89e28a0856b60e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:17:52.288645 2022 raft_consensus.cc:482] T 6d660c1b36c847a7994237b228122339 P 0ec8274e1c6846f2b89e28a0856b60e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:17:52.288692 2022 raft_consensus.cc:3037] T 6d660c1b36c847a7994237b228122339 P 0ec8274e1c6846f2b89e28a0856b60e3 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:17:52.290490 2013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:52.286075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:51103 (local address 127.0.98.65:33939)
0618 22:17:52.286336 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0618 22:17:52.286342 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:17:52.286354 (+ 12us) server_negotiation.cc:407] Connection header received
0618 22:17:52.286395 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:17:52.286398 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:17:52.286437 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:17:52.286501 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:17:52.286809 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:52.287812 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:52.289306 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:52.289861 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:52.290013 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:52.290067 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0618 22:17:52.290323 (+ 256us) server_negotiation.cc:299] Negotiation successful
0618 22:17:52.290383 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":133,"threads_started":1}
I20260618 22:17:52.290503 1885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:52.285364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:33939 (local address 127.0.98.67:51103)
0618 22:17:52.285456 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0618 22:17:52.285471 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:17:52.285563 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:17:52.286559 (+ 996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:17:52.286562 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:17:52.286581 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:17:52.286693 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:52.286698 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:52.287993 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:52.287997 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:52.289167 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:52.289179 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:52.290016 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:52.290020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:52.290177 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:52.290197 (+ 20us) client_negotiation.cc:769] Sending connection context
0618 22:17:52.290299 (+ 102us) client_negotiation.cc:240] Negotiation successful
0618 22:17:52.290373 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"mutex_wait_us":4}
I20260618 22:17:52.290758 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:52.286384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:46989 (local address 127.0.98.66:39177)
0618 22:17:52.286497 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0618 22:17:52.286501 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:17:52.286512 (+ 11us) server_negotiation.cc:407] Connection header received
0618 22:17:52.286544 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:17:52.286547 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:17:52.286613 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:17:52.286679 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:17:52.287473 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:52.288201 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:52.289721 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:52.290235 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:52.290295 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:52.290331 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0618 22:17:52.290580 (+ 249us) server_negotiation.cc:299] Negotiation successful
0618 22:17:52.290642 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260618 22:17:52.290894 2025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:52.285651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:39177 (local address 127.0.98.67:46989)
0618 22:17:52.286177 (+ 526us) negotiation.cc:107] Waiting for socket to connect
0618 22:17:52.286192 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:17:52.286278 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:17:52.286764 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:17:52.286769 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:17:52.286787 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:17:52.287337 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:52.287347 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:52.288349 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:52.288352 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:52.289572 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:52.289601 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:52.290402 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:52.290404 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:52.290473 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:52.290486 (+ 13us) client_negotiation.cc:769] Sending connection context
0618 22:17:52.290566 (+ 80us) client_negotiation.cc:240] Negotiation successful
0618 22:17:52.290740 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":125,"threads_started":1}
I20260618 22:17:52.291256 1666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d660c1b36c847a7994237b228122339" candidate_uuid: "71d4b767e7a841c584be081a7f4a3dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92e3dccb82484cc18b97acc131cd87e7" is_pre_election: true
I20260618 22:17:52.291379 1666 raft_consensus.cc:2454] T 6d660c1b36c847a7994237b228122339 P 92e3dccb82484cc18b97acc131cd87e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 71d4b767e7a841c584be081a7f4a3dbb in term 0.
I20260618 22:17:52.291391 1806 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d660c1b36c847a7994237b228122339" candidate_uuid: "71d4b767e7a841c584be081a7f4a3dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ec8274e1c6846f2b89e28a0856b60e3" is_pre_election: true
I20260618 22:17:52.291728 1878 leader_election.cc:302] T 6d660c1b36c847a7994237b228122339 P 71d4b767e7a841c584be081a7f4a3dbb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71d4b767e7a841c584be081a7f4a3dbb, 92e3dccb82484cc18b97acc131cd87e7; no voters:
I20260618 22:17:52.291949 2023 raft_consensus.cc:2781] T 6d660c1b36c847a7994237b228122339 P 71d4b767e7a841c584be081a7f4a3dbb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:17:52.292050 2023 raft_consensus.cc:482] T 6d660c1b36c847a7994237b228122339 P 71d4b767e7a841c584be081a7f4a3dbb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:17:52.292107 2023 raft_consensus.cc:3037] T 6d660c1b36c847a7994237b228122339 P 71d4b767e7a841c584be081a7f4a3dbb [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:17:52.296559 2023 raft_consensus.cc:504] T 6d660c1b36c847a7994237b228122339 P 71d4b767e7a841c584be081a7f4a3dbb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d4b767e7a841c584be081a7f4a3dbb" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34949 } } peers { permanent_uuid: "92e3dccb82484cc18b97acc131cd87e7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33939 } } peers { permanent_uuid: "0ec8274e1c6846f2b89e28a0856b60e3" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39177 } }
I20260618 22:17:52.296574 2022 raft_consensus.cc:504] T 6d660c1b36c847a7994237b228122339 P 0ec8274e1c6846f2b89e28a0856b60e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d4b767e7a841c584be081a7f4a3dbb" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34949 } } peers { permanent_uuid: "92e3dccb82484cc18b97acc131cd87e7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33939 } } peers { permanent_uuid: "0ec8274e1c6846f2b89e28a0856b60e3" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39177 } }
I20260618 22:17:52.296846 1806 raft_consensus.cc:2379] T 6d660c1b36c847a7994237b228122339 P 0ec8274e1c6846f2b89e28a0856b60e3 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 71d4b767e7a841c584be081a7f4a3dbb in current term 1: Already voted for candidate 0ec8274e1c6846f2b89e28a0856b60e3 in this term.
I20260618 22:17:52.296980 2022 leader_election.cc:288] T 6d660c1b36c847a7994237b228122339 P 0ec8274e1c6846f2b89e28a0856b60e3 [CANDIDATE]: Term 1 election: Requested vote from peers 71d4b767e7a841c584be081a7f4a3dbb (127.0.98.67:34949), 92e3dccb82484cc18b97acc131cd87e7 (127.0.98.65:33939)
I20260618 22:17:52.297042 2023 leader_election.cc:288] T 6d660c1b36c847a7994237b228122339 P 71d4b767e7a841c584be081a7f4a3dbb [CANDIDATE]: Term 1 election: Requested vote from peers 92e3dccb82484cc18b97acc131cd87e7 (127.0.98.65:33939), 0ec8274e1c6846f2b89e28a0856b60e3 (127.0.98.66:39177)
I20260618 22:17:52.297509 1931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d660c1b36c847a7994237b228122339" candidate_uuid: "0ec8274e1c6846f2b89e28a0856b60e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71d4b767e7a841c584be081a7f4a3dbb"
I20260618 22:17:52.297699 1931 raft_consensus.cc:2379] T 6d660c1b36c847a7994237b228122339 P 71d4b767e7a841c584be081a7f4a3dbb [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 0ec8274e1c6846f2b89e28a0856b60e3 in current term 1: Already voted for candidate 71d4b767e7a841c584be081a7f4a3dbb in this term.
I20260618 22:17:52.297713 1802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d660c1b36c847a7994237b228122339" candidate_uuid: "71d4b767e7a841c584be081a7f4a3dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ec8274e1c6846f2b89e28a0856b60e3"
I20260618 22:17:52.297798 1666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d660c1b36c847a7994237b228122339" candidate_uuid: "71d4b767e7a841c584be081a7f4a3dbb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92e3dccb82484cc18b97acc131cd87e7"
I20260618 22:17:52.297844 1802 raft_consensus.cc:2379] T 6d660c1b36c847a7994237b228122339 P 0ec8274e1c6846f2b89e28a0856b60e3 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 71d4b767e7a841c584be081a7f4a3dbb in current term 1: Already voted for candidate 0ec8274e1c6846f2b89e28a0856b60e3 in this term.
I20260618 22:17:52.297924 1666 raft_consensus.cc:3037] T 6d660c1b36c847a7994237b228122339 P 92e3dccb82484cc18b97acc131cd87e7 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:17:52.298568 1665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d660c1b36c847a7994237b228122339" candidate_uuid: "0ec8274e1c6846f2b89e28a0856b60e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92e3dccb82484cc18b97acc131cd87e7"
I20260618 22:17:52.299057 1738 leader_election.cc:302] T 6d660c1b36c847a7994237b228122339 P 0ec8274e1c6846f2b89e28a0856b60e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0ec8274e1c6846f2b89e28a0856b60e3; no voters: 71d4b767e7a841c584be081a7f4a3dbb, 92e3dccb82484cc18b97acc131cd87e7
I20260618 22:17:52.299224 1666 raft_consensus.cc:2454] T 6d660c1b36c847a7994237b228122339 P 92e3dccb82484cc18b97acc131cd87e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 71d4b767e7a841c584be081a7f4a3dbb in term 1.
I20260618 22:17:52.299295 2022 raft_consensus.cc:2726] T 6d660c1b36c847a7994237b228122339 P 0ec8274e1c6846f2b89e28a0856b60e3 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260618 22:17:52.299482 1878 leader_election.cc:302] T 6d660c1b36c847a7994237b228122339 P 71d4b767e7a841c584be081a7f4a3dbb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 71d4b767e7a841c584be081a7f4a3dbb, 92e3dccb82484cc18b97acc131cd87e7; no voters: 0ec8274e1c6846f2b89e28a0856b60e3
I20260618 22:17:52.299726 2023 raft_consensus.cc:2781] T 6d660c1b36c847a7994237b228122339 P 71d4b767e7a841c584be081a7f4a3dbb [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:17:52.299930 2023 raft_consensus.cc:686] T 6d660c1b36c847a7994237b228122339 P 71d4b767e7a841c584be081a7f4a3dbb [term 1 LEADER]: Becoming Leader. State: Replica: 71d4b767e7a841c584be081a7f4a3dbb, State: Running, Role: LEADER
I20260618 22:17:52.300259 2023 consensus_queue.cc:237] T 6d660c1b36c847a7994237b228122339 P 71d4b767e7a841c584be081a7f4a3dbb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d4b767e7a841c584be081a7f4a3dbb" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34949 } } peers { permanent_uuid: "92e3dccb82484cc18b97acc131cd87e7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33939 } } peers { permanent_uuid: "0ec8274e1c6846f2b89e28a0856b60e3" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39177 } }
I20260618 22:17:52.303297 1508 catalog_manager.cc:5495] T 6d660c1b36c847a7994237b228122339 P 71d4b767e7a841c584be081a7f4a3dbb reported cstate change: term changed from 0 to 1, leader changed from <none> to 71d4b767e7a841c584be081a7f4a3dbb (127.0.98.67). New cstate: current_term: 1 leader_uuid: "71d4b767e7a841c584be081a7f4a3dbb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71d4b767e7a841c584be081a7f4a3dbb" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92e3dccb82484cc18b97acc131cd87e7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ec8274e1c6846f2b89e28a0856b60e3" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39177 } health_report { overall_health: UNKNOWN } } }
W20260618 22:17:52.313982 1993 tablet.cc:2369] T 6d660c1b36c847a7994237b228122339 P 71d4b767e7a841c584be081a7f4a3dbb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260618 22:17:52.358146 1853 tablet.cc:2369] T 6d660c1b36c847a7994237b228122339 P 0ec8274e1c6846f2b89e28a0856b60e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:17:52.359923 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:52.355976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51630 (local address 127.0.98.67:34949)
0618 22:17:52.356092 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0618 22:17:52.356096 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:17:52.356110 (+ 14us) server_negotiation.cc:407] Connection header received
0618 22:17:52.356288 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:17:52.356290 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:17:52.356341 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:17:52.356428 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:17:52.356672 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:52.357220 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:52.358422 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:52.358741 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:52.358829 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:52.359162 (+ 333us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:17:52.359528 (+ 366us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:17:52.359611 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0618 22:17:52.359729 (+ 118us) server_negotiation.cc:299] Negotiation successful
0618 22:17:52.359777 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
W20260618 22:17:52.361490 1713 tablet.cc:2369] T 6d660c1b36c847a7994237b228122339 P 92e3dccb82484cc18b97acc131cd87e7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:17:52.365567 1666 raft_consensus.cc:1261] T 6d660c1b36c847a7994237b228122339 P 92e3dccb82484cc18b97acc131cd87e7 [term 1 FOLLOWER]: Refusing update from remote peer 71d4b767e7a841c584be081a7f4a3dbb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:17:52.365609 1802 raft_consensus.cc:1261] T 6d660c1b36c847a7994237b228122339 P 0ec8274e1c6846f2b89e28a0856b60e3 [term 1 FOLLOWER]: Refusing update from remote peer 71d4b767e7a841c584be081a7f4a3dbb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:17:52.366130 2023 consensus_queue.cc:1035] T 6d660c1b36c847a7994237b228122339 P 71d4b767e7a841c584be081a7f4a3dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ec8274e1c6846f2b89e28a0856b60e3" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:17:52.366381 2027 consensus_queue.cc:1035] T 6d660c1b36c847a7994237b228122339 P 71d4b767e7a841c584be081a7f4a3dbb [LEADER]: Connected to new peer: Peer: permanent_uuid: "92e3dccb82484cc18b97acc131cd87e7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:17:52.374202 2030 mvcc.cc:205] Tried to move back new op lower bound from 7298339112403009536 to 7298339112144519168. Current Snapshot: MvccSnapshot[applied={T|T < 7298339112403009536}]
I20260618 22:17:52.376891 2031 mvcc.cc:205] Tried to move back new op lower bound from 7298339112403009536 to 7298339112144519168. Current Snapshot: MvccSnapshot[applied={T|T < 7298339112403009536}]
I20260618 22:17:52.385169 1507 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56076:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:17:52.385322 1507 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56076:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:17:52.387979 1507 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 36253638f49349559f799f12de5095b6: Sending DeleteTablet for 3 replicas of tablet 6d660c1b36c847a7994237b228122339
I20260618 22:17:52.388689 1926 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d660c1b36c847a7994237b228122339 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:17:52 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:51628
I20260618 22:17:52.388921 1646 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d660c1b36c847a7994237b228122339 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:17:52 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:45820
I20260618 22:17:52.388916 1786 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d660c1b36c847a7994237b228122339 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:17:52 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:60282
I20260618 22:17:52.389314 2040 tablet_replica.cc:330] T 6d660c1b36c847a7994237b228122339 P 71d4b767e7a841c584be081a7f4a3dbb: stopping tablet replica
I20260618 22:17:52.389535 2042 tablet_replica.cc:330] T 6d660c1b36c847a7994237b228122339 P 92e3dccb82484cc18b97acc131cd87e7: stopping tablet replica
I20260618 22:17:52.389802 2042 raft_consensus.cc:2229] T 6d660c1b36c847a7994237b228122339 P 92e3dccb82484cc18b97acc131cd87e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:17:52.390117 2042 raft_consensus.cc:2258] T 6d660c1b36c847a7994237b228122339 P 92e3dccb82484cc18b97acc131cd87e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:17:52.390323 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56076:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260618 22:17:52.390488 2041 tablet_replica.cc:330] T 6d660c1b36c847a7994237b228122339 P 0ec8274e1c6846f2b89e28a0856b60e3: stopping tablet replica
W20260618 22:17:52.390698 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:17:52.390743 2041 raft_consensus.cc:2229] T 6d660c1b36c847a7994237b228122339 P 0ec8274e1c6846f2b89e28a0856b60e3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:17:52.391106 2041 raft_consensus.cc:2258] T 6d660c1b36c847a7994237b228122339 P 0ec8274e1c6846f2b89e28a0856b60e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:17:52.391475 2040 raft_consensus.cc:2229] T 6d660c1b36c847a7994237b228122339 P 71d4b767e7a841c584be081a7f4a3dbb [term 1 LEADER]: Raft consensus shutting down.
I20260618 22:17:52.391849 2040 raft_consensus.cc:2258] T 6d660c1b36c847a7994237b228122339 P 71d4b767e7a841c584be081a7f4a3dbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:17:52.392670 2042 ts_tablet_manager.cc:1905] T 6d660c1b36c847a7994237b228122339 P 92e3dccb82484cc18b97acc131cd87e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:17:52.394357 2040 ts_tablet_manager.cc:1905] T 6d660c1b36c847a7994237b228122339 P 71d4b767e7a841c584be081a7f4a3dbb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:17:52.395061 2041 ts_tablet_manager.cc:1905] T 6d660c1b36c847a7994237b228122339 P 0ec8274e1c6846f2b89e28a0856b60e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:17:52.395056 1646 tablet_service.cc:1467] Processing CreateTablet for tablet 552f6565958d4962959cfcfe176e41cb (DEFAULT_TABLE table=test-table [id=3beee996e5ff4b55a0050392394f2346]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:17:52.395179 1786 tablet_service.cc:1467] Processing CreateTablet for tablet 552f6565958d4962959cfcfe176e41cb (DEFAULT_TABLE table=test-table [id=3beee996e5ff4b55a0050392394f2346]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:17:52.395310 1646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 552f6565958d4962959cfcfe176e41cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:17:52.395407 1786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 552f6565958d4962959cfcfe176e41cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:17:52.397446 2041 ts_tablet_manager.cc:1918] T 6d660c1b36c847a7994237b228122339 P 0ec8274e1c6846f2b89e28a0856b60e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:17:52.397806 2015 tablet_bootstrap.cc:492] T 552f6565958d4962959cfcfe176e41cb P 92e3dccb82484cc18b97acc131cd87e7: Bootstrap starting.
I20260618 22:17:52.398731 2015 tablet_bootstrap.cc:654] T 552f6565958d4962959cfcfe176e41cb P 92e3dccb82484cc18b97acc131cd87e7: Neither blocks nor log segments found. Creating new log.
I20260618 22:17:52.399325 2042 ts_tablet_manager.cc:1918] T 6d660c1b36c847a7994237b228122339 P 92e3dccb82484cc18b97acc131cd87e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:17:52.399427 2042 log.cc:1196] T 6d660c1b36c847a7994237b228122339 P 92e3dccb82484cc18b97acc131cd87e7: Deleting WAL directory at /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/minicluster-data/ts-0/wal/wals/6d660c1b36c847a7994237b228122339
I20260618 22:17:52.399636 2016 tablet_bootstrap.cc:492] T 552f6565958d4962959cfcfe176e41cb P 0ec8274e1c6846f2b89e28a0856b60e3: Bootstrap starting.
I20260618 22:17:52.399752 2042 ts_tablet_manager.cc:1939] T 6d660c1b36c847a7994237b228122339 P 92e3dccb82484cc18b97acc131cd87e7: Deleting consensus metadata
I20260618 22:17:52.400488 2016 tablet_bootstrap.cc:654] T 552f6565958d4962959cfcfe176e41cb P 0ec8274e1c6846f2b89e28a0856b60e3: Neither blocks nor log segments found. Creating new log.
I20260618 22:17:52.400622 2041 log.cc:1196] T 6d660c1b36c847a7994237b228122339 P 0ec8274e1c6846f2b89e28a0856b60e3: Deleting WAL directory at /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/minicluster-data/ts-1/wal/wals/6d660c1b36c847a7994237b228122339
I20260618 22:17:52.400869 2041 ts_tablet_manager.cc:1939] T 6d660c1b36c847a7994237b228122339 P 0ec8274e1c6846f2b89e28a0856b60e3: Deleting consensus metadata
I20260618 22:17:52.401281 1926 tablet_service.cc:1467] Processing CreateTablet for tablet 552f6565958d4962959cfcfe176e41cb (DEFAULT_TABLE table=test-table [id=3beee996e5ff4b55a0050392394f2346]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:17:52.401513 1926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 552f6565958d4962959cfcfe176e41cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:17:52.401841 1493 catalog_manager.cc:4841] TS 0ec8274e1c6846f2b89e28a0856b60e3 (127.0.98.66:39177): tablet 6d660c1b36c847a7994237b228122339 (table test-table [id=1daf54b54dca4924a8b57abf9c61fcae]) successfully deleted
I20260618 22:17:52.403699 2017 tablet_bootstrap.cc:492] T 552f6565958d4962959cfcfe176e41cb P 71d4b767e7a841c584be081a7f4a3dbb: Bootstrap starting.
I20260618 22:17:52.404325 1494 catalog_manager.cc:4841] TS 92e3dccb82484cc18b97acc131cd87e7 (127.0.98.65:33939): tablet 6d660c1b36c847a7994237b228122339 (table test-table [id=1daf54b54dca4924a8b57abf9c61fcae]) successfully deleted
I20260618 22:17:52.404654 2017 tablet_bootstrap.cc:654] T 552f6565958d4962959cfcfe176e41cb P 71d4b767e7a841c584be081a7f4a3dbb: Neither blocks nor log segments found. Creating new log.
I20260618 22:17:52.404891 2016 tablet_bootstrap.cc:492] T 552f6565958d4962959cfcfe176e41cb P 0ec8274e1c6846f2b89e28a0856b60e3: No bootstrap required, opened a new log
I20260618 22:17:52.404973 2016 ts_tablet_manager.cc:1397] T 552f6565958d4962959cfcfe176e41cb P 0ec8274e1c6846f2b89e28a0856b60e3: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260618 22:17:52.405370 2016 raft_consensus.cc:348] T 552f6565958d4962959cfcfe176e41cb P 0ec8274e1c6846f2b89e28a0856b60e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e3dccb82484cc18b97acc131cd87e7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33939 } } peers { permanent_uuid: "0ec8274e1c6846f2b89e28a0856b60e3" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39177 } } peers { permanent_uuid: "71d4b767e7a841c584be081a7f4a3dbb" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34949 } }
I20260618 22:17:52.405483 2016 raft_consensus.cc:374] T 552f6565958d4962959cfcfe176e41cb P 0ec8274e1c6846f2b89e28a0856b60e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:17:52.405520 2016 raft_consensus.cc:729] T 552f6565958d4962959cfcfe176e41cb P 0ec8274e1c6846f2b89e28a0856b60e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ec8274e1c6846f2b89e28a0856b60e3, State: Initialized, Role: FOLLOWER
I20260618 22:17:52.405627 2016 consensus_queue.cc:260] T 552f6565958d4962959cfcfe176e41cb P 0ec8274e1c6846f2b89e28a0856b60e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e3dccb82484cc18b97acc131cd87e7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33939 } } peers { permanent_uuid: "0ec8274e1c6846f2b89e28a0856b60e3" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39177 } } peers { permanent_uuid: "71d4b767e7a841c584be081a7f4a3dbb" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34949 } }
I20260618 22:17:52.405809 2017 tablet_bootstrap.cc:492] T 552f6565958d4962959cfcfe176e41cb P 71d4b767e7a841c584be081a7f4a3dbb: No bootstrap required, opened a new log
I20260618 22:17:52.405857 2016 ts_tablet_manager.cc:1428] T 552f6565958d4962959cfcfe176e41cb P 0ec8274e1c6846f2b89e28a0856b60e3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260618 22:17:52.405879 2017 ts_tablet_manager.cc:1397] T 552f6565958d4962959cfcfe176e41cb P 71d4b767e7a841c584be081a7f4a3dbb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260618 22:17:52.406320 2017 raft_consensus.cc:348] T 552f6565958d4962959cfcfe176e41cb P 71d4b767e7a841c584be081a7f4a3dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e3dccb82484cc18b97acc131cd87e7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33939 } } peers { permanent_uuid: "0ec8274e1c6846f2b89e28a0856b60e3" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39177 } } peers { permanent_uuid: "71d4b767e7a841c584be081a7f4a3dbb" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34949 } }
I20260618 22:17:52.406431 2017 raft_consensus.cc:374] T 552f6565958d4962959cfcfe176e41cb P 71d4b767e7a841c584be081a7f4a3dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:17:52.406481 2017 raft_consensus.cc:729] T 552f6565958d4962959cfcfe176e41cb P 71d4b767e7a841c584be081a7f4a3dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71d4b767e7a841c584be081a7f4a3dbb, State: Initialized, Role: FOLLOWER
I20260618 22:17:52.406586 2015 tablet_bootstrap.cc:492] T 552f6565958d4962959cfcfe176e41cb P 92e3dccb82484cc18b97acc131cd87e7: No bootstrap required, opened a new log
I20260618 22:17:52.406595 2017 consensus_queue.cc:260] T 552f6565958d4962959cfcfe176e41cb P 71d4b767e7a841c584be081a7f4a3dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e3dccb82484cc18b97acc131cd87e7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33939 } } peers { permanent_uuid: "0ec8274e1c6846f2b89e28a0856b60e3" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39177 } } peers { permanent_uuid: "71d4b767e7a841c584be081a7f4a3dbb" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34949 } }
I20260618 22:17:52.406661 2015 ts_tablet_manager.cc:1397] T 552f6565958d4962959cfcfe176e41cb P 92e3dccb82484cc18b97acc131cd87e7: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260618 22:17:52.406843 2017 ts_tablet_manager.cc:1428] T 552f6565958d4962959cfcfe176e41cb P 71d4b767e7a841c584be081a7f4a3dbb: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260618 22:17:52.407039 2015 raft_consensus.cc:348] T 552f6565958d4962959cfcfe176e41cb P 92e3dccb82484cc18b97acc131cd87e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e3dccb82484cc18b97acc131cd87e7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33939 } } peers { permanent_uuid: "0ec8274e1c6846f2b89e28a0856b60e3" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39177 } } peers { permanent_uuid: "71d4b767e7a841c584be081a7f4a3dbb" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34949 } }
I20260618 22:17:52.407148 2015 raft_consensus.cc:374] T 552f6565958d4962959cfcfe176e41cb P 92e3dccb82484cc18b97acc131cd87e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:17:52.407184 2015 raft_consensus.cc:729] T 552f6565958d4962959cfcfe176e41cb P 92e3dccb82484cc18b97acc131cd87e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92e3dccb82484cc18b97acc131cd87e7, State: Initialized, Role: FOLLOWER
I20260618 22:17:52.407296 2015 consensus_queue.cc:260] T 552f6565958d4962959cfcfe176e41cb P 92e3dccb82484cc18b97acc131cd87e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e3dccb82484cc18b97acc131cd87e7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33939 } } peers { permanent_uuid: "0ec8274e1c6846f2b89e28a0856b60e3" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39177 } } peers { permanent_uuid: "71d4b767e7a841c584be081a7f4a3dbb" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34949 } }
I20260618 22:17:52.410020 2015 ts_tablet_manager.cc:1428] T 552f6565958d4962959cfcfe176e41cb P 92e3dccb82484cc18b97acc131cd87e7: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20260618 22:17:52.414063 2040 ts_tablet_manager.cc:1918] T 6d660c1b36c847a7994237b228122339 P 71d4b767e7a841c584be081a7f4a3dbb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:17:52.414158 2040 log.cc:1196] T 6d660c1b36c847a7994237b228122339 P 71d4b767e7a841c584be081a7f4a3dbb: Deleting WAL directory at /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/minicluster-data/ts-2/wal/wals/6d660c1b36c847a7994237b228122339
I20260618 22:17:52.414464 2040 ts_tablet_manager.cc:1939] T 6d660c1b36c847a7994237b228122339 P 71d4b767e7a841c584be081a7f4a3dbb: Deleting consensus metadata
I20260618 22:17:52.415503 1495 catalog_manager.cc:4841] TS 71d4b767e7a841c584be081a7f4a3dbb (127.0.98.67:34949): tablet 6d660c1b36c847a7994237b228122339 (table test-table [id=1daf54b54dca4924a8b57abf9c61fcae]) successfully deleted
I20260618 22:17:52.503676 2022 raft_consensus.cc:482] T 552f6565958d4962959cfcfe176e41cb P 0ec8274e1c6846f2b89e28a0856b60e3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:17:52.503818 2022 raft_consensus.cc:504] T 552f6565958d4962959cfcfe176e41cb P 0ec8274e1c6846f2b89e28a0856b60e3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e3dccb82484cc18b97acc131cd87e7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33939 } } peers { permanent_uuid: "0ec8274e1c6846f2b89e28a0856b60e3" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39177 } } peers { permanent_uuid: "71d4b767e7a841c584be081a7f4a3dbb" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34949 } }
I20260618 22:17:52.504299 2022 leader_election.cc:288] T 552f6565958d4962959cfcfe176e41cb P 0ec8274e1c6846f2b89e28a0856b60e3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 92e3dccb82484cc18b97acc131cd87e7 (127.0.98.65:33939), 71d4b767e7a841c584be081a7f4a3dbb (127.0.98.67:34949)
I20260618 22:17:52.504683 1666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "552f6565958d4962959cfcfe176e41cb" candidate_uuid: "0ec8274e1c6846f2b89e28a0856b60e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92e3dccb82484cc18b97acc131cd87e7" is_pre_election: true
I20260618 22:17:52.504686 1931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "552f6565958d4962959cfcfe176e41cb" candidate_uuid: "0ec8274e1c6846f2b89e28a0856b60e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71d4b767e7a841c584be081a7f4a3dbb" is_pre_election: true
I20260618 22:17:52.504839 1666 raft_consensus.cc:2454] T 552f6565958d4962959cfcfe176e41cb P 92e3dccb82484cc18b97acc131cd87e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ec8274e1c6846f2b89e28a0856b60e3 in term 0.
I20260618 22:17:52.504858 1931 raft_consensus.cc:2454] T 552f6565958d4962959cfcfe176e41cb P 71d4b767e7a841c584be081a7f4a3dbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ec8274e1c6846f2b89e28a0856b60e3 in term 0.
I20260618 22:17:52.505162 1738 leader_election.cc:302] T 552f6565958d4962959cfcfe176e41cb P 0ec8274e1c6846f2b89e28a0856b60e3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ec8274e1c6846f2b89e28a0856b60e3, 92e3dccb82484cc18b97acc131cd87e7; no voters:
I20260618 22:17:52.505365 2022 raft_consensus.cc:2781] T 552f6565958d4962959cfcfe176e41cb P 0ec8274e1c6846f2b89e28a0856b60e3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:17:52.505420 2022 raft_consensus.cc:482] T 552f6565958d4962959cfcfe176e41cb P 0ec8274e1c6846f2b89e28a0856b60e3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:17:52.505465 2022 raft_consensus.cc:3037] T 552f6565958d4962959cfcfe176e41cb P 0ec8274e1c6846f2b89e28a0856b60e3 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:17:52.506726 2022 raft_consensus.cc:504] T 552f6565958d4962959cfcfe176e41cb P 0ec8274e1c6846f2b89e28a0856b60e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e3dccb82484cc18b97acc131cd87e7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33939 } } peers { permanent_uuid: "0ec8274e1c6846f2b89e28a0856b60e3" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39177 } } peers { permanent_uuid: "71d4b767e7a841c584be081a7f4a3dbb" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34949 } }
I20260618 22:17:52.507068 2022 leader_election.cc:288] T 552f6565958d4962959cfcfe176e41cb P 0ec8274e1c6846f2b89e28a0856b60e3 [CANDIDATE]: Term 1 election: Requested vote from peers 92e3dccb82484cc18b97acc131cd87e7 (127.0.98.65:33939), 71d4b767e7a841c584be081a7f4a3dbb (127.0.98.67:34949)
I20260618 22:17:52.507403 1931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "552f6565958d4962959cfcfe176e41cb" candidate_uuid: "0ec8274e1c6846f2b89e28a0856b60e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71d4b767e7a841c584be081a7f4a3dbb"
I20260618 22:17:52.507526 1931 raft_consensus.cc:3037] T 552f6565958d4962959cfcfe176e41cb P 71d4b767e7a841c584be081a7f4a3dbb [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:17:52.508680 1931 raft_consensus.cc:2454] T 552f6565958d4962959cfcfe176e41cb P 71d4b767e7a841c584be081a7f4a3dbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ec8274e1c6846f2b89e28a0856b60e3 in term 1.
I20260618 22:17:52.508911 1666 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "552f6565958d4962959cfcfe176e41cb" candidate_uuid: "0ec8274e1c6846f2b89e28a0856b60e3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92e3dccb82484cc18b97acc131cd87e7"
I20260618 22:17:52.508968 1739 leader_election.cc:302] T 552f6565958d4962959cfcfe176e41cb P 0ec8274e1c6846f2b89e28a0856b60e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ec8274e1c6846f2b89e28a0856b60e3, 71d4b767e7a841c584be081a7f4a3dbb; no voters:
I20260618 22:17:52.509028 1666 raft_consensus.cc:3037] T 552f6565958d4962959cfcfe176e41cb P 92e3dccb82484cc18b97acc131cd87e7 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:17:52.509171 2022 raft_consensus.cc:2781] T 552f6565958d4962959cfcfe176e41cb P 0ec8274e1c6846f2b89e28a0856b60e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:17:52.509352 2022 raft_consensus.cc:686] T 552f6565958d4962959cfcfe176e41cb P 0ec8274e1c6846f2b89e28a0856b60e3 [term 1 LEADER]: Becoming Leader. State: Replica: 0ec8274e1c6846f2b89e28a0856b60e3, State: Running, Role: LEADER
I20260618 22:17:52.509577 2022 consensus_queue.cc:237] T 552f6565958d4962959cfcfe176e41cb P 0ec8274e1c6846f2b89e28a0856b60e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e3dccb82484cc18b97acc131cd87e7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33939 } } peers { permanent_uuid: "0ec8274e1c6846f2b89e28a0856b60e3" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39177 } } peers { permanent_uuid: "71d4b767e7a841c584be081a7f4a3dbb" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34949 } }
I20260618 22:17:52.510138 1666 raft_consensus.cc:2454] T 552f6565958d4962959cfcfe176e41cb P 92e3dccb82484cc18b97acc131cd87e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ec8274e1c6846f2b89e28a0856b60e3 in term 1.
I20260618 22:17:52.511734 1507 catalog_manager.cc:5495] T 552f6565958d4962959cfcfe176e41cb P 0ec8274e1c6846f2b89e28a0856b60e3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0ec8274e1c6846f2b89e28a0856b60e3 (127.0.98.66). New cstate: current_term: 1 leader_uuid: "0ec8274e1c6846f2b89e28a0856b60e3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e3dccb82484cc18b97acc131cd87e7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33939 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0ec8274e1c6846f2b89e28a0856b60e3" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39177 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "71d4b767e7a841c584be081a7f4a3dbb" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34949 } health_report { overall_health: UNKNOWN } } }
Jun 18 22:17:52 dist-test-slave-g6r4 krb5kdc[1464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821071, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:17:52.559921 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:52.551608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:40163 (local address 127.0.0.1:56080)
0618 22:17:52.551911 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0618 22:17:52.551927 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 22:17:52.552032 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:17:52.552239 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:17:52.552242 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:17:52.552599 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:17:52.552695 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:52.552701 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:52.553490 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:52.553492 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:52.554113 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:52.554121 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:52.554622 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:52.554626 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:52.554700 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:52.555187 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:17:52.555198 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:17:52.556590 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:17:52.558513 (+ 1923us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:17:52.558522 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:17:52.558528 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:17:52.558791 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:17:52.559041 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:17:52.559043 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:17:52.559044 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:17:52.559122 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:17:52.559436 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:17:52.559443 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:17:52.559545 (+ 102us) client_negotiation.cc:769] Sending connection context
0618 22:17:52.559652 (+ 107us) client_negotiation.cc:240] Negotiation successful
0618 22:17:52.559768 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":151,"threads_started":1}
I20260618 22:17:52.559927 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:52.551642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56080 (local address 127.0.98.126:40163)
0618 22:17:52.551759 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0618 22:17:52.551763 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:17:52.551999 (+ 236us) server_negotiation.cc:407] Connection header received
0618 22:17:52.552098 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:17:52.552100 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:17:52.552136 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:17:52.552227 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:17:52.552800 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:52.553392 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:52.554259 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:52.554458 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:52.554536 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:52.556725 (+ 2189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:17:52.556728 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:17:52.556730 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:17:52.556750 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:17:52.558399 (+ 1649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:17:52.558895 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:17:52.558898 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:17:52.558899 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:17:52.558966 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:17:52.559194 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:17:52.559196 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:17:52.559197 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:17:52.559355 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:17:52.559427 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0618 22:17:52.559708 (+ 281us) server_negotiation.cc:299] Negotiation successful
0618 22:17:52.559816 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":16,"server-negotiator.queue_time_us":41}
I20260618 22:17:52.563721 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:56080:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260618 22:17:52.564340 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:17:52.568804 1646 tablet_service.cc:1467] Processing CreateTablet for tablet 5acac3d15668472fa139f128cc1af533 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=079c398f925d4f948d95a12d39cfe6fd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260618 22:17:52.568977 1926 tablet_service.cc:1467] Processing CreateTablet for tablet 5acac3d15668472fa139f128cc1af533 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=079c398f925d4f948d95a12d39cfe6fd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260618 22:17:52.569269 1646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5acac3d15668472fa139f128cc1af533. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:17:52.569379 1926 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5acac3d15668472fa139f128cc1af533. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:17:52.571683 2015 tablet_bootstrap.cc:492] T 5acac3d15668472fa139f128cc1af533 P 92e3dccb82484cc18b97acc131cd87e7: Bootstrap starting.
I20260618 22:17:52.572237 1786 tablet_service.cc:1467] Processing CreateTablet for tablet 5acac3d15668472fa139f128cc1af533 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=079c398f925d4f948d95a12d39cfe6fd]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260618 22:17:52.572570 1786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5acac3d15668472fa139f128cc1af533. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:17:52.572757 2015 tablet_bootstrap.cc:654] T 5acac3d15668472fa139f128cc1af533 P 92e3dccb82484cc18b97acc131cd87e7: Neither blocks nor log segments found. Creating new log.
I20260618 22:17:52.573772 2015 tablet_bootstrap.cc:492] T 5acac3d15668472fa139f128cc1af533 P 92e3dccb82484cc18b97acc131cd87e7: No bootstrap required, opened a new log
I20260618 22:17:52.573838 2015 ts_tablet_manager.cc:1397] T 5acac3d15668472fa139f128cc1af533 P 92e3dccb82484cc18b97acc131cd87e7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260618 22:17:52.574026 2017 tablet_bootstrap.cc:492] T 5acac3d15668472fa139f128cc1af533 P 71d4b767e7a841c584be081a7f4a3dbb: Bootstrap starting.
I20260618 22:17:52.574286 2015 raft_consensus.cc:348] T 5acac3d15668472fa139f128cc1af533 P 92e3dccb82484cc18b97acc131cd87e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e3dccb82484cc18b97acc131cd87e7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33939 } } peers { permanent_uuid: "0ec8274e1c6846f2b89e28a0856b60e3" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39177 } } peers { permanent_uuid: "71d4b767e7a841c584be081a7f4a3dbb" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34949 } }
I20260618 22:17:52.574402 2015 raft_consensus.cc:374] T 5acac3d15668472fa139f128cc1af533 P 92e3dccb82484cc18b97acc131cd87e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:17:52.574445 2015 raft_consensus.cc:729] T 5acac3d15668472fa139f128cc1af533 P 92e3dccb82484cc18b97acc131cd87e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92e3dccb82484cc18b97acc131cd87e7, State: Initialized, Role: FOLLOWER
I20260618 22:17:52.574563 2015 consensus_queue.cc:260] T 5acac3d15668472fa139f128cc1af533 P 92e3dccb82484cc18b97acc131cd87e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e3dccb82484cc18b97acc131cd87e7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33939 } } peers { permanent_uuid: "0ec8274e1c6846f2b89e28a0856b60e3" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39177 } } peers { permanent_uuid: "71d4b767e7a841c584be081a7f4a3dbb" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34949 } }
I20260618 22:17:52.574872 2017 tablet_bootstrap.cc:654] T 5acac3d15668472fa139f128cc1af533 P 71d4b767e7a841c584be081a7f4a3dbb: Neither blocks nor log segments found. Creating new log.
I20260618 22:17:52.574874 2015 ts_tablet_manager.cc:1428] T 5acac3d15668472fa139f128cc1af533 P 92e3dccb82484cc18b97acc131cd87e7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260618 22:17:52.574786 2021 tablet_replica.cc:439] T 5acac3d15668472fa139f128cc1af533 P 92e3dccb82484cc18b97acc131cd87e7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e3dccb82484cc18b97acc131cd87e7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33939 } } peers { permanent_uuid: "0ec8274e1c6846f2b89e28a0856b60e3" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39177 } } peers { permanent_uuid: "71d4b767e7a841c584be081a7f4a3dbb" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34949 } } }
I20260618 22:17:52.574947 2021 tablet_replica.cc:442] T 5acac3d15668472fa139f128cc1af533 P 92e3dccb82484cc18b97acc131cd87e7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 22:17:52.575099 2016 tablet_bootstrap.cc:492] T 5acac3d15668472fa139f128cc1af533 P 0ec8274e1c6846f2b89e28a0856b60e3: Bootstrap starting.
I20260618 22:17:52.576015 2017 tablet_bootstrap.cc:492] T 5acac3d15668472fa139f128cc1af533 P 71d4b767e7a841c584be081a7f4a3dbb: No bootstrap required, opened a new log
I20260618 22:17:52.576025 2016 tablet_bootstrap.cc:654] T 5acac3d15668472fa139f128cc1af533 P 0ec8274e1c6846f2b89e28a0856b60e3: Neither blocks nor log segments found. Creating new log.
I20260618 22:17:52.576100 2017 ts_tablet_manager.cc:1397] T 5acac3d15668472fa139f128cc1af533 P 71d4b767e7a841c584be081a7f4a3dbb: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260618 22:17:52.576539 2017 raft_consensus.cc:348] T 5acac3d15668472fa139f128cc1af533 P 71d4b767e7a841c584be081a7f4a3dbb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e3dccb82484cc18b97acc131cd87e7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33939 } } peers { permanent_uuid: "0ec8274e1c6846f2b89e28a0856b60e3" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39177 } } peers { permanent_uuid: "71d4b767e7a841c584be081a7f4a3dbb" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34949 } }
I20260618 22:17:52.576642 2017 raft_consensus.cc:374] T 5acac3d15668472fa139f128cc1af533 P 71d4b767e7a841c584be081a7f4a3dbb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:17:52.576668 2017 raft_consensus.cc:729] T 5acac3d15668472fa139f128cc1af533 P 71d4b767e7a841c584be081a7f4a3dbb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71d4b767e7a841c584be081a7f4a3dbb, State: Initialized, Role: FOLLOWER
I20260618 22:17:52.576807 2017 consensus_queue.cc:260] T 5acac3d15668472fa139f128cc1af533 P 71d4b767e7a841c584be081a7f4a3dbb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e3dccb82484cc18b97acc131cd87e7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33939 } } peers { permanent_uuid: "0ec8274e1c6846f2b89e28a0856b60e3" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39177 } } peers { permanent_uuid: "71d4b767e7a841c584be081a7f4a3dbb" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34949 } }
I20260618 22:17:52.577085 2017 ts_tablet_manager.cc:1428] T 5acac3d15668472fa139f128cc1af533 P 71d4b767e7a841c584be081a7f4a3dbb: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260618 22:17:52.577049 2037 tablet_replica.cc:439] T 5acac3d15668472fa139f128cc1af533 P 71d4b767e7a841c584be081a7f4a3dbb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e3dccb82484cc18b97acc131cd87e7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33939 } } peers { permanent_uuid: "0ec8274e1c6846f2b89e28a0856b60e3" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39177 } } peers { permanent_uuid: "71d4b767e7a841c584be081a7f4a3dbb" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34949 } } }
I20260618 22:17:52.577183 2037 tablet_replica.cc:442] T 5acac3d15668472fa139f128cc1af533 P 71d4b767e7a841c584be081a7f4a3dbb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 22:17:52.577209 2016 tablet_bootstrap.cc:492] T 5acac3d15668472fa139f128cc1af533 P 0ec8274e1c6846f2b89e28a0856b60e3: No bootstrap required, opened a new log
I20260618 22:17:52.577282 2016 ts_tablet_manager.cc:1397] T 5acac3d15668472fa139f128cc1af533 P 0ec8274e1c6846f2b89e28a0856b60e3: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260618 22:17:52.577720 2016 raft_consensus.cc:348] T 5acac3d15668472fa139f128cc1af533 P 0ec8274e1c6846f2b89e28a0856b60e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e3dccb82484cc18b97acc131cd87e7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33939 } } peers { permanent_uuid: "0ec8274e1c6846f2b89e28a0856b60e3" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39177 } } peers { permanent_uuid: "71d4b767e7a841c584be081a7f4a3dbb" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34949 } }
I20260618 22:17:52.577806 2016 raft_consensus.cc:374] T 5acac3d15668472fa139f128cc1af533 P 0ec8274e1c6846f2b89e28a0856b60e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:17:52.577827 2016 raft_consensus.cc:729] T 5acac3d15668472fa139f128cc1af533 P 0ec8274e1c6846f2b89e28a0856b60e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ec8274e1c6846f2b89e28a0856b60e3, State: Initialized, Role: FOLLOWER
I20260618 22:17:52.577980 2016 consensus_queue.cc:260] T 5acac3d15668472fa139f128cc1af533 P 0ec8274e1c6846f2b89e28a0856b60e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e3dccb82484cc18b97acc131cd87e7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33939 } } peers { permanent_uuid: "0ec8274e1c6846f2b89e28a0856b60e3" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39177 } } peers { permanent_uuid: "71d4b767e7a841c584be081a7f4a3dbb" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34949 } }
I20260618 22:17:52.578230 2016 ts_tablet_manager.cc:1428] T 5acac3d15668472fa139f128cc1af533 P 0ec8274e1c6846f2b89e28a0856b60e3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260618 22:17:52.578184 2022 tablet_replica.cc:439] T 5acac3d15668472fa139f128cc1af533 P 0ec8274e1c6846f2b89e28a0856b60e3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e3dccb82484cc18b97acc131cd87e7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33939 } } peers { permanent_uuid: "0ec8274e1c6846f2b89e28a0856b60e3" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39177 } } peers { permanent_uuid: "71d4b767e7a841c584be081a7f4a3dbb" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34949 } } }
I20260618 22:17:52.578322 2022 tablet_replica.cc:442] T 5acac3d15668472fa139f128cc1af533 P 0ec8274e1c6846f2b89e28a0856b60e3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 22:17:52.786382 2021 raft_consensus.cc:482] T 5acac3d15668472fa139f128cc1af533 P 92e3dccb82484cc18b97acc131cd87e7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:17:52.786588 2021 raft_consensus.cc:504] T 5acac3d15668472fa139f128cc1af533 P 92e3dccb82484cc18b97acc131cd87e7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e3dccb82484cc18b97acc131cd87e7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33939 } } peers { permanent_uuid: "0ec8274e1c6846f2b89e28a0856b60e3" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39177 } } peers { permanent_uuid: "71d4b767e7a841c584be081a7f4a3dbb" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34949 } }
I20260618 22:17:52.787786 2021 leader_election.cc:288] T 5acac3d15668472fa139f128cc1af533 P 92e3dccb82484cc18b97acc131cd87e7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ec8274e1c6846f2b89e28a0856b60e3 (127.0.98.66:39177), 71d4b767e7a841c584be081a7f4a3dbb (127.0.98.67:34949)
I20260618 22:17:52.792506 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:52.788139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:48409 (local address 127.0.98.67:34949)
0618 22:17:52.788365 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0618 22:17:52.788369 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:17:52.788444 (+ 75us) server_negotiation.cc:407] Connection header received
0618 22:17:52.788619 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:17:52.788622 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:17:52.788679 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:17:52.788760 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:17:52.789019 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:52.789586 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:52.791221 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:52.791850 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:52.791926 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:52.791989 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0618 22:17:52.792287 (+ 298us) server_negotiation.cc:299] Negotiation successful
0618 22:17:52.792381 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148}
I20260618 22:17:52.792511 2058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:52.788046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:34949 (local address 127.0.98.65:48409)
0618 22:17:52.788338 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0618 22:17:52.788357 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0618 22:17:52.788505 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:17:52.788775 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:17:52.788778 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:17:52.788794 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:17:52.788915 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:52.788921 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:52.789688 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:52.789695 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:52.791054 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:52.791070 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:52.792079 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:52.792082 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:52.792167 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:52.792189 (+ 22us) client_negotiation.cc:769] Sending connection context
0618 22:17:52.792273 (+ 84us) client_negotiation.cc:240] Negotiation successful
0618 22:17:52.792353 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":44,"thread_start_us":84,"threads_started":1}
I20260618 22:17:52.792780 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:52.788139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:57947 (local address 127.0.98.66:39177)
0618 22:17:52.788293 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0618 22:17:52.788298 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:17:52.788612 (+ 314us) server_negotiation.cc:407] Connection header received
0618 22:17:52.788745 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:17:52.788748 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:17:52.788797 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:17:52.788872 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:17:52.789113 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:52.789654 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:52.791687 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:52.792254 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:52.792345 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:52.792397 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0618 22:17:52.792593 (+ 196us) server_negotiation.cc:299] Negotiation successful
0618 22:17:52.792667 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260618 22:17:52.792860 2057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:52.788046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:39177 (local address 127.0.98.65:57947)
0618 22:17:52.788509 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0618 22:17:52.788520 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 22:17:52.788643 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:17:52.788894 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:17:52.788896 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:17:52.788909 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:17:52.789000 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:52.789007 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:52.789783 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:52.789785 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:52.791064 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:52.791086 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:52.792376 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:52.792380 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:52.792466 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:52.792488 (+ 22us) client_negotiation.cc:769] Sending connection context
0618 22:17:52.792566 (+ 78us) client_negotiation.cc:240] Negotiation successful
0618 22:17:52.792688 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"mutex_wait_us":133,"thread_start_us":90,"threads_started":1}
I20260618 22:17:52.793180 1931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5acac3d15668472fa139f128cc1af533" candidate_uuid: "92e3dccb82484cc18b97acc131cd87e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71d4b767e7a841c584be081a7f4a3dbb" is_pre_election: true
I20260618 22:17:52.793203 1802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5acac3d15668472fa139f128cc1af533" candidate_uuid: "92e3dccb82484cc18b97acc131cd87e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ec8274e1c6846f2b89e28a0856b60e3" is_pre_election: true
I20260618 22:17:52.793330 1931 raft_consensus.cc:2454] T 5acac3d15668472fa139f128cc1af533 P 71d4b767e7a841c584be081a7f4a3dbb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92e3dccb82484cc18b97acc131cd87e7 in term 0.
I20260618 22:17:52.793344 1802 raft_consensus.cc:2454] T 5acac3d15668472fa139f128cc1af533 P 0ec8274e1c6846f2b89e28a0856b60e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92e3dccb82484cc18b97acc131cd87e7 in term 0.
I20260618 22:17:52.793678 1599 leader_election.cc:302] T 5acac3d15668472fa139f128cc1af533 P 92e3dccb82484cc18b97acc131cd87e7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71d4b767e7a841c584be081a7f4a3dbb, 92e3dccb82484cc18b97acc131cd87e7; no voters:
I20260618 22:17:52.793936 2021 raft_consensus.cc:2781] T 5acac3d15668472fa139f128cc1af533 P 92e3dccb82484cc18b97acc131cd87e7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:17:52.794018 2021 raft_consensus.cc:482] T 5acac3d15668472fa139f128cc1af533 P 92e3dccb82484cc18b97acc131cd87e7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:17:52.794071 2021 raft_consensus.cc:3037] T 5acac3d15668472fa139f128cc1af533 P 92e3dccb82484cc18b97acc131cd87e7 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:17:52.794993 2021 raft_consensus.cc:504] T 5acac3d15668472fa139f128cc1af533 P 92e3dccb82484cc18b97acc131cd87e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e3dccb82484cc18b97acc131cd87e7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33939 } } peers { permanent_uuid: "0ec8274e1c6846f2b89e28a0856b60e3" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39177 } } peers { permanent_uuid: "71d4b767e7a841c584be081a7f4a3dbb" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34949 } }
I20260618 22:17:52.795328 2021 leader_election.cc:288] T 5acac3d15668472fa139f128cc1af533 P 92e3dccb82484cc18b97acc131cd87e7 [CANDIDATE]: Term 1 election: Requested vote from peers 0ec8274e1c6846f2b89e28a0856b60e3 (127.0.98.66:39177), 71d4b767e7a841c584be081a7f4a3dbb (127.0.98.67:34949)
I20260618 22:17:52.795672 1802 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5acac3d15668472fa139f128cc1af533" candidate_uuid: "92e3dccb82484cc18b97acc131cd87e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ec8274e1c6846f2b89e28a0856b60e3"
I20260618 22:17:52.795770 1802 raft_consensus.cc:3037] T 5acac3d15668472fa139f128cc1af533 P 0ec8274e1c6846f2b89e28a0856b60e3 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:17:52.796661 1802 raft_consensus.cc:2454] T 5acac3d15668472fa139f128cc1af533 P 0ec8274e1c6846f2b89e28a0856b60e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92e3dccb82484cc18b97acc131cd87e7 in term 1.
I20260618 22:17:52.796898 1931 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5acac3d15668472fa139f128cc1af533" candidate_uuid: "92e3dccb82484cc18b97acc131cd87e7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71d4b767e7a841c584be081a7f4a3dbb"
I20260618 22:17:52.796942 1597 leader_election.cc:302] T 5acac3d15668472fa139f128cc1af533 P 92e3dccb82484cc18b97acc131cd87e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ec8274e1c6846f2b89e28a0856b60e3, 92e3dccb82484cc18b97acc131cd87e7; no voters:
I20260618 22:17:52.797013 1931 raft_consensus.cc:3037] T 5acac3d15668472fa139f128cc1af533 P 71d4b767e7a841c584be081a7f4a3dbb [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:17:52.797106 2021 raft_consensus.cc:2781] T 5acac3d15668472fa139f128cc1af533 P 92e3dccb82484cc18b97acc131cd87e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:17:52.797287 2021 raft_consensus.cc:686] T 5acac3d15668472fa139f128cc1af533 P 92e3dccb82484cc18b97acc131cd87e7 [term 1 LEADER]: Becoming Leader. State: Replica: 92e3dccb82484cc18b97acc131cd87e7, State: Running, Role: LEADER
I20260618 22:17:52.797742 2021 consensus_queue.cc:237] T 5acac3d15668472fa139f128cc1af533 P 92e3dccb82484cc18b97acc131cd87e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e3dccb82484cc18b97acc131cd87e7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33939 } } peers { permanent_uuid: "0ec8274e1c6846f2b89e28a0856b60e3" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39177 } } peers { permanent_uuid: "71d4b767e7a841c584be081a7f4a3dbb" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34949 } }
I20260618 22:17:52.798159 1931 raft_consensus.cc:2454] T 5acac3d15668472fa139f128cc1af533 P 71d4b767e7a841c584be081a7f4a3dbb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92e3dccb82484cc18b97acc131cd87e7 in term 1.
I20260618 22:17:52.799688 2059 tablet_replica.cc:439] T 5acac3d15668472fa139f128cc1af533 P 92e3dccb82484cc18b97acc131cd87e7: TxnStatusTablet state changed. Reason: New leader 92e3dccb82484cc18b97acc131cd87e7. Latest consensus state: current_term: 1 leader_uuid: "92e3dccb82484cc18b97acc131cd87e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e3dccb82484cc18b97acc131cd87e7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33939 } } peers { permanent_uuid: "0ec8274e1c6846f2b89e28a0856b60e3" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39177 } } peers { permanent_uuid: "71d4b767e7a841c584be081a7f4a3dbb" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34949 } } }
I20260618 22:17:52.799803 2059 tablet_replica.cc:442] T 5acac3d15668472fa139f128cc1af533 P 92e3dccb82484cc18b97acc131cd87e7: This TxnStatusTablet replica's current role is: LEADER
I20260618 22:17:52.800195 2061 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260618 22:17:52.800357 2061 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260618 22:17:52.800978 1508 catalog_manager.cc:5495] T 5acac3d15668472fa139f128cc1af533 P 92e3dccb82484cc18b97acc131cd87e7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 92e3dccb82484cc18b97acc131cd87e7 (127.0.98.65). New cstate: current_term: 1 leader_uuid: "92e3dccb82484cc18b97acc131cd87e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e3dccb82484cc18b97acc131cd87e7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33939 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0ec8274e1c6846f2b89e28a0856b60e3" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39177 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "71d4b767e7a841c584be081a7f4a3dbb" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34949 } health_report { overall_health: UNKNOWN } } }
I20260618 22:17:52.815953 2053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:52.812168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:33939 (local address 127.0.0.1:45832)
0618 22:17:52.812283 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0618 22:17:52.812300 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0618 22:17:52.812405 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:17:52.812718 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:17:52.812721 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:17:52.812730 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0618 22:17:52.812836 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:52.812842 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:52.813707 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:52.813712 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:52.814470 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:52.814479 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:52.814916 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:52.814919 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:52.815010 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:52.815053 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0618 22:17:52.815667 (+ 614us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0618 22:17:52.815680 (+ 13us) client_negotiation.cc:769] Sending connection context
0618 22:17:52.815761 (+ 81us) client_negotiation.cc:240] Negotiation successful
0618 22:17:52.815843 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260618 22:17:52.815968 2064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:52.812231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45832 (local address 127.0.98.65:33939)
0618 22:17:52.812461 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0618 22:17:52.812472 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0618 22:17:52.812486 (+ 14us) server_negotiation.cc:407] Connection header received
0618 22:17:52.812542 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:17:52.812544 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:17:52.812598 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:17:52.812704 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:17:52.812952 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:52.813607 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:52.814577 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:52.814815 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:52.814917 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:52.815187 (+ 270us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:17:52.815566 (+ 379us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:17:52.815679 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0618 22:17:52.815773 (+ 94us) server_negotiation.cc:299] Negotiation successful
0618 22:17:52.815820 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":82,"threads_started":1}
I20260618 22:17:52.818455 1802 raft_consensus.cc:1261] T 5acac3d15668472fa139f128cc1af533 P 0ec8274e1c6846f2b89e28a0856b60e3 [term 1 FOLLOWER]: Refusing update from remote peer 92e3dccb82484cc18b97acc131cd87e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:17:52.818583 1931 raft_consensus.cc:1261] T 5acac3d15668472fa139f128cc1af533 P 71d4b767e7a841c584be081a7f4a3dbb [term 1 FOLLOWER]: Refusing update from remote peer 92e3dccb82484cc18b97acc131cd87e7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:17:52.818913 2021 consensus_queue.cc:1035] T 5acac3d15668472fa139f128cc1af533 P 92e3dccb82484cc18b97acc131cd87e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ec8274e1c6846f2b89e28a0856b60e3" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39177 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:17:52.819074 2059 consensus_queue.cc:1035] T 5acac3d15668472fa139f128cc1af533 P 92e3dccb82484cc18b97acc131cd87e7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71d4b767e7a841c584be081a7f4a3dbb" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:17:52.820569 2037 tablet_replica.cc:439] T 5acac3d15668472fa139f128cc1af533 P 71d4b767e7a841c584be081a7f4a3dbb: TxnStatusTablet state changed. Reason: New leader 92e3dccb82484cc18b97acc131cd87e7. Latest consensus state: current_term: 1 leader_uuid: "92e3dccb82484cc18b97acc131cd87e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e3dccb82484cc18b97acc131cd87e7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33939 } } peers { permanent_uuid: "0ec8274e1c6846f2b89e28a0856b60e3" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39177 } } peers { permanent_uuid: "71d4b767e7a841c584be081a7f4a3dbb" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34949 } } }
I20260618 22:17:52.820673 2037 tablet_replica.cc:442] T 5acac3d15668472fa139f128cc1af533 P 71d4b767e7a841c584be081a7f4a3dbb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 22:17:52.820757 2022 tablet_replica.cc:439] T 5acac3d15668472fa139f128cc1af533 P 0ec8274e1c6846f2b89e28a0856b60e3: TxnStatusTablet state changed. Reason: New leader 92e3dccb82484cc18b97acc131cd87e7. Latest consensus state: current_term: 1 leader_uuid: "92e3dccb82484cc18b97acc131cd87e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e3dccb82484cc18b97acc131cd87e7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33939 } } peers { permanent_uuid: "0ec8274e1c6846f2b89e28a0856b60e3" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39177 } } peers { permanent_uuid: "71d4b767e7a841c584be081a7f4a3dbb" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34949 } } }
I20260618 22:17:52.820855 2022 tablet_replica.cc:442] T 5acac3d15668472fa139f128cc1af533 P 0ec8274e1c6846f2b89e28a0856b60e3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 22:17:52.822477 2059 tablet_replica.cc:439] T 5acac3d15668472fa139f128cc1af533 P 92e3dccb82484cc18b97acc131cd87e7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "92e3dccb82484cc18b97acc131cd87e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e3dccb82484cc18b97acc131cd87e7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33939 } } peers { permanent_uuid: "0ec8274e1c6846f2b89e28a0856b60e3" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39177 } } peers { permanent_uuid: "71d4b767e7a841c584be081a7f4a3dbb" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34949 } } }
I20260618 22:17:52.822574 2059 tablet_replica.cc:442] T 5acac3d15668472fa139f128cc1af533 P 92e3dccb82484cc18b97acc131cd87e7: This TxnStatusTablet replica's current role is: LEADER
I20260618 22:17:52.822671 2059 tablet_replica.cc:439] T 5acac3d15668472fa139f128cc1af533 P 92e3dccb82484cc18b97acc131cd87e7: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "92e3dccb82484cc18b97acc131cd87e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e3dccb82484cc18b97acc131cd87e7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33939 } } peers { permanent_uuid: "0ec8274e1c6846f2b89e28a0856b60e3" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39177 } } peers { permanent_uuid: "71d4b767e7a841c584be081a7f4a3dbb" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34949 } } }
I20260618 22:17:52.822731 2059 tablet_replica.cc:442] T 5acac3d15668472fa139f128cc1af533 P 92e3dccb82484cc18b97acc131cd87e7: This TxnStatusTablet replica's current role is: LEADER
I20260618 22:17:52.823967 2037 tablet_replica.cc:439] T 5acac3d15668472fa139f128cc1af533 P 71d4b767e7a841c584be081a7f4a3dbb: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "92e3dccb82484cc18b97acc131cd87e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e3dccb82484cc18b97acc131cd87e7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33939 } } peers { permanent_uuid: "0ec8274e1c6846f2b89e28a0856b60e3" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39177 } } peers { permanent_uuid: "71d4b767e7a841c584be081a7f4a3dbb" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34949 } } }
I20260618 22:17:52.824071 2037 tablet_replica.cc:442] T 5acac3d15668472fa139f128cc1af533 P 71d4b767e7a841c584be081a7f4a3dbb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 22:17:52.825886 2022 tablet_replica.cc:439] T 5acac3d15668472fa139f128cc1af533 P 0ec8274e1c6846f2b89e28a0856b60e3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "92e3dccb82484cc18b97acc131cd87e7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92e3dccb82484cc18b97acc131cd87e7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33939 } } peers { permanent_uuid: "0ec8274e1c6846f2b89e28a0856b60e3" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39177 } } peers { permanent_uuid: "71d4b767e7a841c584be081a7f4a3dbb" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34949 } } }
I20260618 22:17:52.826016 2022 tablet_replica.cc:442] T 5acac3d15668472fa139f128cc1af533 P 0ec8274e1c6846f2b89e28a0856b60e3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 22:17:52.831941 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:52.828314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60294 (local address 127.0.98.66:39177)
0618 22:17:52.828442 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0618 22:17:52.828447 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:17:52.828465 (+ 18us) server_negotiation.cc:407] Connection header received
0618 22:17:52.828584 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:17:52.828586 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:17:52.828628 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:17:52.828743 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:17:52.828946 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:52.829490 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:52.830670 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:52.830889 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:52.830965 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:52.831230 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:17:52.831608 (+ 378us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:17:52.831684 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0618 22:17:52.831769 (+ 85us) server_negotiation.cc:299] Negotiation successful
0618 22:17:52.831809 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260618 22:17:52.841490 2064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:52.838440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:59895 (local address 127.0.98.65:33939)
0618 22:17:52.838622 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0618 22:17:52.838629 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:17:52.838667 (+ 38us) server_negotiation.cc:407] Connection header received
0618 22:17:52.838837 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:17:52.838839 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:17:52.838892 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:17:52.838977 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:17:52.839194 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:52.839675 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:52.840508 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:52.840738 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:52.840840 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:52.841099 (+ 259us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:17:52.841164 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:17:52.841244 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0618 22:17:52.841322 (+ 78us) server_negotiation.cc:299] Negotiation successful
0618 22:17:52.841367 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260618 22:17:52.841518 2069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:52.838340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:33939 (local address 127.0.98.66:59895)
0618 22:17:52.838599 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0618 22:17:52.838613 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:17:52.838712 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:17:52.838988 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:17:52.838991 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:17:52.838999 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0618 22:17:52.839090 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:52.839096 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:52.839786 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:52.839789 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:52.840389 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:52.840397 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:52.840841 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:52.840845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:52.840926 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:52.840969 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0618 22:17:52.841242 (+ 273us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0618 22:17:52.841248 (+ 6us) client_negotiation.cc:769] Sending connection context
0618 22:17:52.841312 (+ 64us) client_negotiation.cc:240] Negotiation successful
0618 22:17:52.841383 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":101,"threads_started":1}
I20260618 22:17:52.847580 1931 raft_consensus.cc:1261] T 552f6565958d4962959cfcfe176e41cb P 71d4b767e7a841c584be081a7f4a3dbb [term 1 FOLLOWER]: Refusing update from remote peer 0ec8274e1c6846f2b89e28a0856b60e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:17:52.847630 1666 raft_consensus.cc:1261] T 552f6565958d4962959cfcfe176e41cb P 92e3dccb82484cc18b97acc131cd87e7 [term 1 FOLLOWER]: Refusing update from remote peer 0ec8274e1c6846f2b89e28a0856b60e3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:17:52.848014 2046 consensus_queue.cc:1035] T 552f6565958d4962959cfcfe176e41cb P 0ec8274e1c6846f2b89e28a0856b60e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71d4b767e7a841c584be081a7f4a3dbb" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:17:52.848309 2046 consensus_queue.cc:1035] T 552f6565958d4962959cfcfe176e41cb P 0ec8274e1c6846f2b89e28a0856b60e3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92e3dccb82484cc18b97acc131cd87e7" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:17:52.851469 2032 mvcc.cc:205] Tried to move back new op lower bound from 7298339114380042240 to 7298339113001623552. Current Snapshot: MvccSnapshot[applied={T|T < 7298339114380042240}]
I20260618 22:17:52.881592 2014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:52.874487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:54971 (local address 127.0.98.66:39177)
0618 22:17:52.874589 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0618 22:17:52.874593 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:17:52.877544 (+ 2951us) server_negotiation.cc:407] Connection header received
0618 22:17:52.877731 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:17:52.877734 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:17:52.877787 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:17:52.877831 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:17:52.878244 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:52.879046 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:52.880413 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:52.880738 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:52.880834 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:52.881182 (+ 348us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:17:52.881252 (+ 70us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:17:52.881326 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0618 22:17:52.881415 (+ 89us) server_negotiation.cc:299] Negotiation successful
0618 22:17:52.881460 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260618 22:17:52.881613 2074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:52.873515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:39177 (local address 127.0.98.65:54971)
0618 22:17:52.877468 (+ 3953us) negotiation.cc:107] Waiting for socket to connect
0618 22:17:52.877483 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:17:52.877617 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:17:52.877992 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:17:52.877995 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:17:52.878002 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0618 22:17:52.878128 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:52.878134 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:52.879144 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:52.879147 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:52.880275 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:52.880286 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:52.880916 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:52.880920 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:52.881012 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:52.881056 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0618 22:17:52.881329 (+ 273us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0618 22:17:52.881337 (+ 8us) client_negotiation.cc:769] Sending connection context
0618 22:17:52.881398 (+ 61us) client_negotiation.cc:240] Negotiation successful
0618 22:17:52.881469 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3795,"thread_start_us":104,"threads_started":1}
I20260618 22:17:52.906222 1509 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56076:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:17:52.906363 1509 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56076:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:17:52.908597 1509 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 36253638f49349559f799f12de5095b6: Sending DeleteTablet for 3 replicas of tablet 552f6565958d4962959cfcfe176e41cb
I20260618 22:17:52.909310 1786 tablet_service.cc:1514] Processing DeleteTablet for tablet 552f6565958d4962959cfcfe176e41cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:17:52 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:60282
I20260618 22:17:52.909332 1646 tablet_service.cc:1514] Processing DeleteTablet for tablet 552f6565958d4962959cfcfe176e41cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:17:52 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:45820
I20260618 22:17:52.909813 1926 tablet_service.cc:1514] Processing DeleteTablet for tablet 552f6565958d4962959cfcfe176e41cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:17:52 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:51628
I20260618 22:17:52.910750 2040 tablet_replica.cc:330] T 552f6565958d4962959cfcfe176e41cb P 71d4b767e7a841c584be081a7f4a3dbb: stopping tablet replica
I20260618 22:17:52.910879 2040 raft_consensus.cc:2229] T 552f6565958d4962959cfcfe176e41cb P 71d4b767e7a841c584be081a7f4a3dbb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:17:52.910977 2040 raft_consensus.cc:2258] T 552f6565958d4962959cfcfe176e41cb P 71d4b767e7a841c584be081a7f4a3dbb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:17:52.911691 2040 ts_tablet_manager.cc:1905] T 552f6565958d4962959cfcfe176e41cb P 71d4b767e7a841c584be081a7f4a3dbb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:17:52.914019 2079 tablet_replica.cc:330] T 552f6565958d4962959cfcfe176e41cb P 92e3dccb82484cc18b97acc131cd87e7: stopping tablet replica
I20260618 22:17:52.914129 2079 raft_consensus.cc:2229] T 552f6565958d4962959cfcfe176e41cb P 92e3dccb82484cc18b97acc131cd87e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:17:52.914224 2079 raft_consensus.cc:2258] T 552f6565958d4962959cfcfe176e41cb P 92e3dccb82484cc18b97acc131cd87e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:17:52.914588 2040 ts_tablet_manager.cc:1918] T 552f6565958d4962959cfcfe176e41cb P 71d4b767e7a841c584be081a7f4a3dbb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260618 22:17:52.914651 2040 log.cc:1196] T 552f6565958d4962959cfcfe176e41cb P 71d4b767e7a841c584be081a7f4a3dbb: Deleting WAL directory at /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/minicluster-data/ts-2/wal/wals/552f6565958d4962959cfcfe176e41cb
I20260618 22:17:52.914883 2040 ts_tablet_manager.cc:1939] T 552f6565958d4962959cfcfe176e41cb P 71d4b767e7a841c584be081a7f4a3dbb: Deleting consensus metadata
I20260618 22:17:52.915658 1495 catalog_manager.cc:4841] TS 71d4b767e7a841c584be081a7f4a3dbb (127.0.98.67:34949): tablet 552f6565958d4962959cfcfe176e41cb (table test-table [id=3beee996e5ff4b55a0050392394f2346]) successfully deleted
I20260618 22:17:52.916023 2078 tablet_replica.cc:330] T 552f6565958d4962959cfcfe176e41cb P 0ec8274e1c6846f2b89e28a0856b60e3: stopping tablet replica
I20260618 22:17:52.916200 2078 raft_consensus.cc:2229] T 552f6565958d4962959cfcfe176e41cb P 0ec8274e1c6846f2b89e28a0856b60e3 [term 1 LEADER]: Raft consensus shutting down.
I20260618 22:17:52.916488 2078 raft_consensus.cc:2258] T 552f6565958d4962959cfcfe176e41cb P 0ec8274e1c6846f2b89e28a0856b60e3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:17:52.917127 2078 ts_tablet_manager.cc:1905] T 552f6565958d4962959cfcfe176e41cb P 0ec8274e1c6846f2b89e28a0856b60e3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:17:52.920331 2078 ts_tablet_manager.cc:1918] T 552f6565958d4962959cfcfe176e41cb P 0ec8274e1c6846f2b89e28a0856b60e3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260618 22:17:52.920548 2078 log.cc:1196] T 552f6565958d4962959cfcfe176e41cb P 0ec8274e1c6846f2b89e28a0856b60e3: Deleting WAL directory at /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/minicluster-data/ts-1/wal/wals/552f6565958d4962959cfcfe176e41cb
I20260618 22:17:52.920864 2078 ts_tablet_manager.cc:1939] T 552f6565958d4962959cfcfe176e41cb P 0ec8274e1c6846f2b89e28a0856b60e3: Deleting consensus metadata
Jun 18 22:17:52 dist-test-slave-g6r4 krb5kdc[1464](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821072, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.65@KRBTEST.COM
I20260618 22:17:52.921548 1493 catalog_manager.cc:4841] TS 0ec8274e1c6846f2b89e28a0856b60e3 (127.0.98.66:39177): tablet 552f6565958d4962959cfcfe176e41cb (table test-table [id=3beee996e5ff4b55a0050392394f2346]) successfully deleted
I20260618 22:17:52.925750 2064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:52.910966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45846 (local address 127.0.98.65:33939)
0618 22:17:52.911065 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0618 22:17:52.911069 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:17:52.914196 (+ 3127us) server_negotiation.cc:407] Connection header received
0618 22:17:52.914421 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:17:52.914424 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:17:52.914478 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:17:52.914554 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:17:52.915520 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:52.916314 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:52.917518 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:52.917862 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:52.917962 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:52.921711 (+ 3749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:17:52.921720 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:17:52.921744 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:17:52.921770 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:17:52.923727 (+ 1957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:17:52.924253 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:17:52.924259 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:17:52.924261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:17:52.924365 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:17:52.924698 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:17:52.924702 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:17:52.924703 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:17:52.924973 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:17:52.925182 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0618 22:17:52.925446 (+ 264us) server_negotiation.cc:299] Negotiation successful
0618 22:17:52.925567 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260618 22:17:52.926342 2079 ts_tablet_manager.cc:1905] T 552f6565958d4962959cfcfe176e41cb P 92e3dccb82484cc18b97acc131cd87e7: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260618 22:17:52.926318 1696 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45846
I20260618 22:17:52.929518 2079 ts_tablet_manager.cc:1918] T 552f6565958d4962959cfcfe176e41cb P 92e3dccb82484cc18b97acc131cd87e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260618 22:17:52.929618 2079 log.cc:1196] T 552f6565958d4962959cfcfe176e41cb P 92e3dccb82484cc18b97acc131cd87e7: Deleting WAL directory at /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781821065380026-393-0/minicluster-data/ts-0/wal/wals/552f6565958d4962959cfcfe176e41cb
I20260618 22:17:52.929858 2079 ts_tablet_manager.cc:1939] T 552f6565958d4962959cfcfe176e41cb P 92e3dccb82484cc18b97acc131cd87e7: Deleting consensus metadata
I20260618 22:17:52.930537 1494 catalog_manager.cc:4841] TS 92e3dccb82484cc18b97acc131cd87e7 (127.0.98.65:33939): tablet 552f6565958d4962959cfcfe176e41cb (table test-table [id=3beee996e5ff4b55a0050392394f2346]) successfully deleted
I20260618 22:17:52.937567 1558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:52.928646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56084 (local address 127.0.98.126:40163)
0618 22:17:52.928762 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0618 22:17:52.928766 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:17:52.928946 (+ 180us) server_negotiation.cc:407] Connection header received
0618 22:17:52.929131 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:17:52.929134 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:17:52.929192 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:17:52.929271 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:17:52.930062 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:52.930891 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:52.932029 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:52.932389 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:52.932514 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:52.934157 (+ 1643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:17:52.934161 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:17:52.934164 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:17:52.934189 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:17:52.935945 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:17:52.936389 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:17:52.936393 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:17:52.936396 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:17:52.936476 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:17:52.936741 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:17:52.936744 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:17:52.936746 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:17:52.936939 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:17:52.937077 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0618 22:17:52.937307 (+ 230us) server_negotiation.cc:299] Negotiation successful
0618 22:17:52.937415 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260618 22:17:52.938112 1509 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56084
I20260618 22:17:52.939591 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 1581
I20260618 22:17:52.947250 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 1721
I20260618 22:17:52.953346 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 1861
I20260618 22:17:52.959601 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 1477
2026-06-18T22:17:52Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2925 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:17:53 dist-test-slave-g6r4 krb5kdc[2086](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:17:53 dist-test-slave-g6r4 krb5kdc[2086](info): set up 2 sockets
Jun 18 22:17:53 dist-test-slave-g6r4 krb5kdc[2086](info): commencing operation
krb5kdc: starting...
W20260618 22:17:53.998186 393 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/krb5kdc/test-user.keytab.
Jun 18 22:17:54 dist-test-slave-g6r4 krb5kdc[2086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821074, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:17:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:17:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260618 22:17:54.117784 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:46833
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:46701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:46833
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:17:54.224043 2099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:17:54.224336 2099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:17:54.224421 2099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:17:54.228385 2099 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:17:54.228466 2099 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260618 22:17:54.228499 2099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:17:54.228540 2099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:17:54.228588 2099 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:17:54.228626 2099 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:17:54.233657 2099 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:46701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:46833
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:46833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2099
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:17:54.234649 2099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:17:54.235601 2099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:17:54.244582 2104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:17:54.244578 2107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:17:54.244731 2099 server_base.cc:1034] running on GCE node
W20260618 22:17:54.244582 2105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:17:54.245437 2099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:17:54.246526 2099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:17:54.247678 2099 hybrid_clock.cc:648] HybridClock initialized: now 1781821074247665 us; error 27 us; skew 500 ppm
Jun 18 22:17:54 dist-test-slave-g6r4 krb5kdc[2086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821074, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:17:54.250936 2099 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260618 22:17:54.251763 2099 webserver.cc:460] Webserver started at http://127.0.98.126:39855/ using document root <none> and password file <none>
I20260618 22:17:54.252290 2099 fs_manager.cc:359] Metadata directory not provided
I20260618 22:17:54.252349 2099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:17:54.252549 2099 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:17:54.254519 2099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/master-0/data/instance:
uuid: "c640d89b20c44a6cb8af3a75a11e252a"
format_stamp: "Formatted at 2026-06-18 22:17:54 on dist-test-slave-g6r4"
server_key: "3efd9d16e057f305ee3cb70812098a98"
server_key_iv: "7be57e3ddfc6b9daf677470a95695d72"
server_key_version: "encryptionkey@0"
I20260618 22:17:54.255023 2099 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/master-0/wal/instance:
uuid: "c640d89b20c44a6cb8af3a75a11e252a"
format_stamp: "Formatted at 2026-06-18 22:17:54 on dist-test-slave-g6r4"
server_key: "3efd9d16e057f305ee3cb70812098a98"
server_key_iv: "7be57e3ddfc6b9daf677470a95695d72"
server_key_version: "encryptionkey@0"
I20260618 22:17:54.258656 2099 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:17:54.261314 2114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:17:54.262598 2099 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:17:54.262710 2099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/master-0/wal
uuid: "c640d89b20c44a6cb8af3a75a11e252a"
format_stamp: "Formatted at 2026-06-18 22:17:54 on dist-test-slave-g6r4"
server_key: "3efd9d16e057f305ee3cb70812098a98"
server_key_iv: "7be57e3ddfc6b9daf677470a95695d72"
server_key_version: "encryptionkey@0"
I20260618 22:17:54.262816 2099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:17:54.277829 2099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:17:54.280786 2099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:17:54.280969 2099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:17:54.288635 2099 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:46833
I20260618 22:17:54.288658 2176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:46833 every 8 connection(s)
I20260618 22:17:54.289530 2099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
I20260618 22:17:54.292106 2177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:17:54.293109 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 2099
I20260618 22:17:54.293202 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/master-0/wal/instance
I20260618 22:17:54.293485 393 external_mini_cluster.cc:1428] Setting key 14d7b73cca7dd92fc4169d223823a0b2
I20260618 22:17:54.298506 2177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c640d89b20c44a6cb8af3a75a11e252a: Bootstrap starting.
I20260618 22:17:54.301029 2177 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c640d89b20c44a6cb8af3a75a11e252a: Neither blocks nor log segments found. Creating new log.
I20260618 22:17:54.301734 2177 log.cc:824] T 00000000000000000000000000000000 P c640d89b20c44a6cb8af3a75a11e252a: Log is configured to *not* fsync() on all Append() calls
Jun 18 22:17:54 dist-test-slave-g6r4 krb5kdc[2086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821074, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:17:54.303946 2177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c640d89b20c44a6cb8af3a75a11e252a: No bootstrap required, opened a new log
I20260618 22:17:54.308145 2177 raft_consensus.cc:348] T 00000000000000000000000000000000 P c640d89b20c44a6cb8af3a75a11e252a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c640d89b20c44a6cb8af3a75a11e252a" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46833 } }
I20260618 22:17:54.308408 2177 raft_consensus.cc:374] T 00000000000000000000000000000000 P c640d89b20c44a6cb8af3a75a11e252a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:17:54.308461 2177 raft_consensus.cc:729] T 00000000000000000000000000000000 P c640d89b20c44a6cb8af3a75a11e252a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c640d89b20c44a6cb8af3a75a11e252a, State: Initialized, Role: FOLLOWER
I20260618 22:17:54.308702 2180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:54.294820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38886 (local address 127.0.98.126:46833)
0618 22:17:54.295209 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0618 22:17:54.295230 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0618 22:17:54.295274 (+ 44us) server_negotiation.cc:407] Connection header received
0618 22:17:54.296057 (+ 783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:17:54.296080 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:17:54.296474 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:17:54.296841 (+ 367us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:17:54.297554 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:54.298962 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:54.299782 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:54.300443 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:54.300598 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:54.303194 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:17:54.303200 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:17:54.303215 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:17:54.303244 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:17:54.305725 (+ 2481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:17:54.306173 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:17:54.306179 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:17:54.306186 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:17:54.306323 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:17:54.306566 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:17:54.306569 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:17:54.306571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:17:54.307070 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:17:54.307277 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0618 22:17:54.307464 (+ 187us) server_negotiation.cc:299] Negotiation successful
0618 22:17:54.307763 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":94,"threads_started":1}
I20260618 22:17:54.308923 2177 consensus_queue.cc:260] T 00000000000000000000000000000000 P c640d89b20c44a6cb8af3a75a11e252a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c640d89b20c44a6cb8af3a75a11e252a" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46833 } }
I20260618 22:17:54.309073 2177 raft_consensus.cc:388] T 00000000000000000000000000000000 P c640d89b20c44a6cb8af3a75a11e252a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:17:54.309134 2177 raft_consensus.cc:482] T 00000000000000000000000000000000 P c640d89b20c44a6cb8af3a75a11e252a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:17:54.309233 2177 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c640d89b20c44a6cb8af3a75a11e252a [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:17:54.310288 2177 raft_consensus.cc:504] T 00000000000000000000000000000000 P c640d89b20c44a6cb8af3a75a11e252a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c640d89b20c44a6cb8af3a75a11e252a" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46833 } }
I20260618 22:17:54.310618 2177 leader_election.cc:302] T 00000000000000000000000000000000 P c640d89b20c44a6cb8af3a75a11e252a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c640d89b20c44a6cb8af3a75a11e252a; no voters:
I20260618 22:17:54.310968 2177 leader_election.cc:288] T 00000000000000000000000000000000 P c640d89b20c44a6cb8af3a75a11e252a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:17:54.311084 2182 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c640d89b20c44a6cb8af3a75a11e252a [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:17:54.311321 2182 raft_consensus.cc:686] T 00000000000000000000000000000000 P c640d89b20c44a6cb8af3a75a11e252a [term 1 LEADER]: Becoming Leader. State: Replica: c640d89b20c44a6cb8af3a75a11e252a, State: Running, Role: LEADER
I20260618 22:17:54.311617 2182 consensus_queue.cc:237] T 00000000000000000000000000000000 P c640d89b20c44a6cb8af3a75a11e252a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c640d89b20c44a6cb8af3a75a11e252a" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46833 } }
I20260618 22:17:54.312132 2177 sys_catalog.cc:564] T 00000000000000000000000000000000 P c640d89b20c44a6cb8af3a75a11e252a [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:17:54.313866 2183 sys_catalog.cc:455] T 00000000000000000000000000000000 P c640d89b20c44a6cb8af3a75a11e252a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c640d89b20c44a6cb8af3a75a11e252a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c640d89b20c44a6cb8af3a75a11e252a" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46833 } } }
I20260618 22:17:54.314043 2183 sys_catalog.cc:458] T 00000000000000000000000000000000 P c640d89b20c44a6cb8af3a75a11e252a [sys.catalog]: This master's current role is: LEADER
I20260618 22:17:54.314417 2184 sys_catalog.cc:455] T 00000000000000000000000000000000 P c640d89b20c44a6cb8af3a75a11e252a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c640d89b20c44a6cb8af3a75a11e252a. Latest consensus state: current_term: 1 leader_uuid: "c640d89b20c44a6cb8af3a75a11e252a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c640d89b20c44a6cb8af3a75a11e252a" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46833 } } }
I20260618 22:17:54.314512 2184 sys_catalog.cc:458] T 00000000000000000000000000000000 P c640d89b20c44a6cb8af3a75a11e252a [sys.catalog]: This master's current role is: LEADER
W20260618 22:17:54.316556 2196 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c640d89b20c44a6cb8af3a75a11e252a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 22:17:54.316618 2196 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 22:17:54.316699 2198 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:17:54.317354 2198 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:17:54.322165 2198 catalog_manager.cc:1347] Generated new cluster ID: 0058f4b351f4491794ff9011dfd23e6d
I20260618 22:17:54.322228 2198 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:17:54.344444 2198 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:17:54.345336 2198 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:17:54.351055 2198 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c640d89b20c44a6cb8af3a75a11e252a: Generated new TSK 0
I20260618 22:17:54.351753 2198 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab.
I20260618 22:17:54.387188 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:46833
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:46701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:17:54.494953 2203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:17:54.495175 2203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:17:54.495218 2203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:17:54.499166 2203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:17:54.499217 2203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:17:54.499254 2203 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260618 22:17:54.499269 2203 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260618 22:17:54.499343 2203 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260618 22:17:54.504375 2203 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:46701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.126:46833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2203
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:17:54.505252 2203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:17:54.506201 2203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:17:54.515308 2208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:17:54.515329 2211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:17:54.515394 2209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:17:54.515650 2203 server_base.cc:1034] running on GCE node
I20260618 22:17:54.516040 2203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:17:54.516619 2203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:17:54.517792 2203 hybrid_clock.cc:648] HybridClock initialized: now 1781821074517777 us; error 30 us; skew 500 ppm
Jun 18 22:17:54 dist-test-slave-g6r4 krb5kdc[2086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821074, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:17:54.520587 2203 init.cc:377] Logged in from keytab as kudu/127.0.98.65@KRBTEST.COM (short username kudu)
I20260618 22:17:54.521384 2203 webserver.cc:460] Webserver started at http://127.0.98.65:36197/ using document root <none> and password file <none>
I20260618 22:17:54.521973 2203 fs_manager.cc:359] Metadata directory not provided
I20260618 22:17:54.522056 2203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:17:54.522271 2203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:17:54.524166 2203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/ts-0/data/instance:
uuid: "c5de8b6fc77c4d1090acf81f46f74192"
format_stamp: "Formatted at 2026-06-18 22:17:54 on dist-test-slave-g6r4"
server_key: "a27fcc724345b9e895ca482cae5852c0"
server_key_iv: "9072315853282b77160022d5227e0039"
server_key_version: "encryptionkey@0"
I20260618 22:17:54.524704 2203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/ts-0/wal/instance:
uuid: "c5de8b6fc77c4d1090acf81f46f74192"
format_stamp: "Formatted at 2026-06-18 22:17:54 on dist-test-slave-g6r4"
server_key: "a27fcc724345b9e895ca482cae5852c0"
server_key_iv: "9072315853282b77160022d5227e0039"
server_key_version: "encryptionkey@0"
I20260618 22:17:54.528154 2203 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260618 22:17:54.530413 2218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:17:54.531706 2203 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:17:54.531845 2203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/ts-0/wal
uuid: "c5de8b6fc77c4d1090acf81f46f74192"
format_stamp: "Formatted at 2026-06-18 22:17:54 on dist-test-slave-g6r4"
server_key: "a27fcc724345b9e895ca482cae5852c0"
server_key_iv: "9072315853282b77160022d5227e0039"
server_key_version: "encryptionkey@0"
I20260618 22:17:54.531945 2203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:17:54.544723 2203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:17:54.547289 2203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:17:54.547472 2203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:17:54.548717 2203 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:17:54.548786 2203 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:17:54.548856 2203 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:17:54.548895 2203 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 18 22:17:54 dist-test-slave-g6r4 krb5kdc[2086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821074, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:17:54.560405 2203 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:35281
I20260618 22:17:54.560719 2333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:35281 every 8 connection(s)
I20260618 22:17:54.561421 2203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
I20260618 22:17:54.562780 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 2203
I20260618 22:17:54.562896 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/ts-0/wal/instance
I20260618 22:17:54.563158 2180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:54.549495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:35901 (local address 127.0.98.126:46833)
0618 22:17:54.549604 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0618 22:17:54.549608 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:17:54.550584 (+ 976us) server_negotiation.cc:407] Connection header received
0618 22:17:54.551547 (+ 963us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:17:54.551550 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:17:54.551604 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:17:54.551665 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:17:54.552963 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:54.553842 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:54.555139 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:54.555387 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:54.555494 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:54.558894 (+ 3400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:17:54.558899 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:17:54.558901 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:17:54.558929 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:17:54.560781 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:17:54.561364 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:17:54.561368 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:17:54.561371 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:17:54.561469 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:17:54.561940 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:17:54.561945 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:17:54.561948 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:17:54.562196 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:17:54.562316 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0618 22:17:54.562868 (+ 552us) server_negotiation.cc:299] Negotiation successful
0618 22:17:54.562993 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260618 22:17:54.563184 393 external_mini_cluster.cc:1428] Setting key 8855e658696f93c2bfe06206847278ea
I20260618 22:17:54.563930 2227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:54.550009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:46833 (local address 127.0.98.65:35901)
0618 22:17:54.550452 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0618 22:17:54.550490 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0618 22:17:54.551304 (+ 814us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:17:54.551955 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:17:54.551960 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:17:54.552254 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:17:54.552828 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:54.552844 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:54.554023 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:54.554026 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:54.555017 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:54.555026 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:54.555528 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:54.555532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:54.555687 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:54.556268 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:17:54.556292 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:17:54.558693 (+ 2401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:17:54.560917 (+ 2224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:17:54.560923 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:17:54.560936 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:17:54.561257 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:17:54.561588 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:17:54.561593 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:17:54.561596 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:17:54.561778 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:17:54.562332 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:17:54.562338 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:17:54.562626 (+ 288us) client_negotiation.cc:769] Sending connection context
0618 22:17:54.562814 (+ 188us) client_negotiation.cc:240] Negotiation successful
0618 22:17:54.563106 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":128,"threads_started":1}
I20260618 22:17:54.565127 2334 heartbeater.cc:347] Connected to a master server at 127.0.98.126:46833
I20260618 22:17:54.565291 2334 heartbeater.cc:464] Registering TS with master...
I20260618 22:17:54.565827 2334 heartbeater.cc:511] Master 127.0.98.126:46833 requested a full tablet report, sending...
I20260618 22:17:54.567770 2130 ts_manager.cc:194] Registered new tserver with Master: c5de8b6fc77c4d1090acf81f46f74192 (127.0.98.65:35281)
I20260618 22:17:54.568768 2130 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.65@KRBTEST.COM'} at 127.0.98.65:35901
WARNING: no policy specified for kudu/127.0.98.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.66@KRBTEST.COM" created.
I20260618 22:17:54.577692 2341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:54.570192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:46833 (local address 127.0.98.65:55037)
0618 22:17:54.570462 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0618 22:17:54.570475 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:17:54.570579 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:17:54.570910 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:17:54.570912 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:17:54.571068 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:17:54.571149 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:54.571153 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:54.572382 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:54.572385 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:54.572924 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:54.572931 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:54.573518 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:54.573521 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:54.573585 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:54.574111 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:17:54.574122 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:17:54.574411 (+ 289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:17:54.576089 (+ 1678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:17:54.576094 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:17:54.576096 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:17:54.576366 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:17:54.576633 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:17:54.576636 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:17:54.576638 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:17:54.576718 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:17:54.577171 (+ 453us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:17:54.577174 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:17:54.577268 (+ 94us) client_negotiation.cc:769] Sending connection context
0618 22:17:54.577369 (+ 101us) client_negotiation.cc:240] Negotiation successful
0618 22:17:54.577497 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":107,"threads_started":1}
I20260618 22:17:54.577713 2180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:54.570230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:55037 (local address 127.0.98.126:46833)
0618 22:17:54.570431 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0618 22:17:54.570435 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:17:54.570542 (+ 107us) server_negotiation.cc:407] Connection header received
0618 22:17:54.570708 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:17:54.570714 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:17:54.570757 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:17:54.570804 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:17:54.571255 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:54.572179 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:54.573036 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:54.573407 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:54.573525 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:54.574549 (+ 1024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:17:54.574554 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:17:54.574559 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:17:54.574583 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:17:54.575966 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:17:54.576464 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:17:54.576467 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:17:54.576469 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:17:54.576541 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:17:54.576823 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:17:54.576826 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:17:54.576828 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:17:54.577017 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:17:54.577180 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0618 22:17:54.577430 (+ 250us) server_negotiation.cc:299] Negotiation successful
0618 22:17:54.577584 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab.
I20260618 22:17:54.593067 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:46833
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:46701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:17:54.699563 2343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:17:54.699791 2343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:17:54.699836 2343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:17:54.703717 2343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:17:54.703761 2343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:17:54.703799 2343 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260618 22:17:54.703814 2343 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260618 22:17:54.703845 2343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260618 22:17:54.708778 2343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:46701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.126:46833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2343
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:17:54.709628 2343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:17:54.710525 2343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:17:54.719693 2349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:17:54.719805 2348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:17:54.719951 2343 server_base.cc:1034] running on GCE node
W20260618 22:17:54.719808 2351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:17:54.720429 2343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:17:54.721067 2343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:17:54.722240 2343 hybrid_clock.cc:648] HybridClock initialized: now 1781821074722224 us; error 27 us; skew 500 ppm
Jun 18 22:17:54 dist-test-slave-g6r4 krb5kdc[2086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821074, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:17:54.724819 2343 init.cc:377] Logged in from keytab as kudu/127.0.98.66@KRBTEST.COM (short username kudu)
I20260618 22:17:54.725603 2343 webserver.cc:460] Webserver started at http://127.0.98.66:35509/ using document root <none> and password file <none>
I20260618 22:17:54.726186 2343 fs_manager.cc:359] Metadata directory not provided
I20260618 22:17:54.726269 2343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:17:54.726478 2343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:17:54.728446 2343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/ts-1/data/instance:
uuid: "24b2931f243c419a9d1c7d00c97defac"
format_stamp: "Formatted at 2026-06-18 22:17:54 on dist-test-slave-g6r4"
server_key: "0f58063ec3092f419a9f63fc26fbcdf5"
server_key_iv: "01a6ba0fc80a604e64dc6d626850b82c"
server_key_version: "encryptionkey@0"
I20260618 22:17:54.728979 2343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/ts-1/wal/instance:
uuid: "24b2931f243c419a9d1c7d00c97defac"
format_stamp: "Formatted at 2026-06-18 22:17:54 on dist-test-slave-g6r4"
server_key: "0f58063ec3092f419a9f63fc26fbcdf5"
server_key_iv: "01a6ba0fc80a604e64dc6d626850b82c"
server_key_version: "encryptionkey@0"
I20260618 22:17:54.732836 2343 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260618 22:17:54.735265 2358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:17:54.736414 2343 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260618 22:17:54.736505 2343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/ts-1/wal
uuid: "24b2931f243c419a9d1c7d00c97defac"
format_stamp: "Formatted at 2026-06-18 22:17:54 on dist-test-slave-g6r4"
server_key: "0f58063ec3092f419a9f63fc26fbcdf5"
server_key_iv: "01a6ba0fc80a604e64dc6d626850b82c"
server_key_version: "encryptionkey@0"
I20260618 22:17:54.736581 2343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:17:54.748065 2343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:17:54.750557 2343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:17:54.750691 2343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:17:54.751889 2343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:17:54.751960 2343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:17:54.752029 2343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:17:54.752069 2343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 18 22:17:54 dist-test-slave-g6r4 krb5kdc[2086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821074, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:17:54.762456 2343 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:46229
I20260618 22:17:54.762513 2473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:46229 every 8 connection(s)
I20260618 22:17:54.763579 2343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
I20260618 22:17:54.765687 2180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:54.752680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:49745 (local address 127.0.98.126:46833)
0618 22:17:54.752854 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0618 22:17:54.752860 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:17:54.753552 (+ 692us) server_negotiation.cc:407] Connection header received
0618 22:17:54.754492 (+ 940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:17:54.754495 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:17:54.754545 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:17:54.754640 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:17:54.756210 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:54.757397 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:54.758854 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:54.759274 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:54.759408 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:54.762145 (+ 2737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:17:54.762149 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:17:54.762152 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:17:54.762186 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:17:54.763795 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:17:54.764364 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:17:54.764367 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:17:54.764369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:17:54.764457 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:17:54.764802 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:17:54.764804 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:17:54.764805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:17:54.764971 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:17:54.765045 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0618 22:17:54.765459 (+ 414us) server_negotiation.cc:299] Negotiation successful
0618 22:17:54.765548 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260618 22:17:54.766448 2367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:54.752979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:46833 (local address 127.0.98.66:49745)
0618 22:17:54.753394 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0618 22:17:54.753438 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0618 22:17:54.754301 (+ 863us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:17:54.754928 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:17:54.754939 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:17:54.755431 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:17:54.755988 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:54.756007 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:54.757530 (+ 1523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:54.757534 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:54.758723 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:54.758732 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:54.759429 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:54.759434 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:54.759617 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:54.760163 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:17:54.760178 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:17:54.761954 (+ 1776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:17:54.763901 (+ 1947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:17:54.763905 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:17:54.763915 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:17:54.764258 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:17:54.764547 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:17:54.764549 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:17:54.764551 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:17:54.764725 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:17:54.765049 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:17:54.765053 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:17:54.765265 (+ 212us) client_negotiation.cc:769] Sending connection context
0618 22:17:54.765400 (+ 135us) client_negotiation.cc:240] Negotiation successful
0618 22:17:54.765617 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":119,"threads_started":1}
I20260618 22:17:54.767601 2474 heartbeater.cc:347] Connected to a master server at 127.0.98.126:46833
I20260618 22:17:54.767740 2474 heartbeater.cc:464] Registering TS with master...
I20260618 22:17:54.768173 2474 heartbeater.cc:511] Master 127.0.98.126:46833 requested a full tablet report, sending...
I20260618 22:17:54.769150 2131 ts_manager.cc:194] Registered new tserver with Master: 24b2931f243c419a9d1c7d00c97defac (127.0.98.66:46229)
I20260618 22:17:54.769587 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 2343
I20260618 22:17:54.769697 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/ts-1/wal/instance
I20260618 22:17:54.769925 2131 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.66@KRBTEST.COM'} at 127.0.98.66:49745
I20260618 22:17:54.770002 393 external_mini_cluster.cc:1428] Setting key 25722c14e923056bb0b549d60cd1e7df
I20260618 22:17:54.776834 2180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:54.769560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:48199 (local address 127.0.98.126:46833)
0618 22:17:54.769721 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0618 22:17:54.769725 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:17:54.769968 (+ 243us) server_negotiation.cc:407] Connection header received
0618 22:17:54.770126 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:17:54.770130 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:17:54.770179 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:17:54.770250 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:17:54.770860 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:54.771586 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:54.772462 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:54.772693 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:54.772793 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:54.773794 (+ 1001us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:17:54.773798 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:17:54.773800 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:17:54.773821 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:17:54.775255 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:17:54.775669 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:17:54.775674 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:17:54.775676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:17:54.775753 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:17:54.776023 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:17:54.776027 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:17:54.776029 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:17:54.776220 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:17:54.776359 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0618 22:17:54.776602 (+ 243us) server_negotiation.cc:299] Negotiation successful
0618 22:17:54.776704 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260618 22:17:54.776844 2480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:54.769524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:46833 (local address 127.0.98.66:48199)
0618 22:17:54.769854 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0618 22:17:54.769873 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0618 22:17:54.770007 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:17:54.770356 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:17:54.770359 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:17:54.770573 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:17:54.770734 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:54.770741 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:54.771741 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:54.771744 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:54.772348 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:54.772356 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:54.772804 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:54.772807 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:54.772903 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:54.773373 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:17:54.773391 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:17:54.773663 (+ 272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:17:54.775375 (+ 1712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:17:54.775378 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:17:54.775380 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:17:54.775577 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:17:54.775855 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:17:54.775858 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:17:54.775860 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:17:54.775939 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:17:54.776349 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:17:54.776351 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:17:54.776442 (+ 91us) client_negotiation.cc:769] Sending connection context
0618 22:17:54.776543 (+ 101us) client_negotiation.cc:240] Negotiation successful
0618 22:17:54.776668 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"mutex_wait_us":2,"thread_start_us":77,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab.
I20260618 22:17:54.796438 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:46833
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:46701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:17:54.915014 2483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:17:54.915232 2483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:17:54.915300 2483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:17:54.919806 2483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:17:54.919866 2483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:17:54.919906 2483 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260618 22:17:54.919960 2483 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260618 22:17:54.920013 2483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260618 22:17:54.925175 2483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:46701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.126:46833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:17:54.926200 2483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:17:54.927156 2483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:17:54.936777 2489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:17:54.936792 2491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:17:54.936786 2488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:17:54.937083 2483 server_base.cc:1034] running on GCE node
I20260618 22:17:54.937460 2483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:17:54.938133 2483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:17:54.939291 2483 hybrid_clock.cc:648] HybridClock initialized: now 1781821074939274 us; error 28 us; skew 500 ppm
Jun 18 22:17:54 dist-test-slave-g6r4 krb5kdc[2086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821074, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:17:54.942049 2483 init.cc:377] Logged in from keytab as kudu/127.0.98.67@KRBTEST.COM (short username kudu)
I20260618 22:17:54.942808 2483 webserver.cc:460] Webserver started at http://127.0.98.67:37115/ using document root <none> and password file <none>
I20260618 22:17:54.943301 2483 fs_manager.cc:359] Metadata directory not provided
I20260618 22:17:54.943356 2483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:17:54.943508 2483 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:17:54.945374 2483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/ts-2/data/instance:
uuid: "58a33666811f42fca694ee4b30acb12f"
format_stamp: "Formatted at 2026-06-18 22:17:54 on dist-test-slave-g6r4"
server_key: "5d4f5fc300f787c5d8a5686b30f7219f"
server_key_iv: "1c791d8aa4958ee6c395a90e8a1f0e76"
server_key_version: "encryptionkey@0"
I20260618 22:17:54.945853 2483 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/ts-2/wal/instance:
uuid: "58a33666811f42fca694ee4b30acb12f"
format_stamp: "Formatted at 2026-06-18 22:17:54 on dist-test-slave-g6r4"
server_key: "5d4f5fc300f787c5d8a5686b30f7219f"
server_key_iv: "1c791d8aa4958ee6c395a90e8a1f0e76"
server_key_version: "encryptionkey@0"
I20260618 22:17:54.949304 2483 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260618 22:17:54.951637 2498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:17:54.952845 2483 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260618 22:17:54.952983 2483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/ts-2/wal
uuid: "58a33666811f42fca694ee4b30acb12f"
format_stamp: "Formatted at 2026-06-18 22:17:54 on dist-test-slave-g6r4"
server_key: "5d4f5fc300f787c5d8a5686b30f7219f"
server_key_iv: "1c791d8aa4958ee6c395a90e8a1f0e76"
server_key_version: "encryptionkey@0"
I20260618 22:17:54.953087 2483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:17:54.969877 2483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:17:54.972769 2483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:17:54.972958 2483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:17:54.974407 2483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:17:54.974472 2483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:17:54.974537 2483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:17:54.974576 2483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 18 22:17:54 dist-test-slave-g6r4 krb5kdc[2086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821074, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:17:54.985085 2483 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:33011
I20260618 22:17:54.985146 2613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:33011 every 8 connection(s)
I20260618 22:17:54.986148 2483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
I20260618 22:17:54.987952 2180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:54.975040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:37697 (local address 127.0.98.126:46833)
0618 22:17:54.975172 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0618 22:17:54.975175 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:17:54.976158 (+ 983us) server_negotiation.cc:407] Connection header received
0618 22:17:54.977154 (+ 996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:17:54.977158 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:17:54.977214 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:17:54.977322 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:17:54.978615 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:54.979200 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:54.980407 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:54.980691 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:54.980806 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:54.984285 (+ 3479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:17:54.984288 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:17:54.984291 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:17:54.984313 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:17:54.986010 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:17:54.986570 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:17:54.986576 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:17:54.986578 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:17:54.986674 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:17:54.987051 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:17:54.987058 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:17:54.987063 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:17:54.987242 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:17:54.987320 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0618 22:17:54.987751 (+ 431us) server_negotiation.cc:299] Negotiation successful
0618 22:17:54.987841 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260618 22:17:54.988689 2507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:54.975559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:46833 (local address 127.0.98.67:37697)
0618 22:17:54.976008 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0618 22:17:54.976057 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0618 22:17:54.976945 (+ 888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:17:54.977494 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:17:54.977504 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:17:54.977998 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:17:54.978466 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:54.978479 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:54.979323 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:54.979326 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:54.980283 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:54.980292 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:54.980836 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:54.980840 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:54.981053 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:54.981627 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:17:54.981644 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:17:54.984108 (+ 2464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:17:54.986116 (+ 2008us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:17:54.986121 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:17:54.986131 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:17:54.986443 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:17:54.986764 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:17:54.986767 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:17:54.986769 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:17:54.986936 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:17:54.987346 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:17:54.987350 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:17:54.987544 (+ 194us) client_negotiation.cc:769] Sending connection context
0618 22:17:54.987699 (+ 155us) client_negotiation.cc:240] Negotiation successful
0618 22:17:54.987917 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":123,"threads_started":1}
I20260618 22:17:54.989961 2614 heartbeater.cc:347] Connected to a master server at 127.0.98.126:46833
I20260618 22:17:54.990078 2614 heartbeater.cc:464] Registering TS with master...
I20260618 22:17:54.990509 2614 heartbeater.cc:511] Master 127.0.98.126:46833 requested a full tablet report, sending...
I20260618 22:17:54.991488 2130 ts_manager.cc:194] Registered new tserver with Master: 58a33666811f42fca694ee4b30acb12f (127.0.98.67:33011)
I20260618 22:17:54.992156 2130 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.67@KRBTEST.COM'} at 127.0.98.67:37697
I20260618 22:17:54.992358 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 2483
I20260618 22:17:54.992486 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/ts-2/wal/instance
I20260618 22:17:54.992789 393 external_mini_cluster.cc:1428] Setting key 776575e92addadeff28f42411add0bb5
I20260618 22:17:54.994879 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 22:17:55.001129 2180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:54.992054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:52347 (local address 127.0.98.126:46833)
0618 22:17:54.992171 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0618 22:17:54.992175 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:17:54.993215 (+ 1040us) server_negotiation.cc:407] Connection header received
0618 22:17:54.993623 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:17:54.993626 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:17:54.993668 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:17:54.993736 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:17:54.994316 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:54.994909 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:54.995762 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:54.996219 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:54.996351 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:54.997624 (+ 1273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:17:54.997631 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:17:54.997636 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:17:54.997665 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:17:54.999373 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:17:54.999816 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:17:54.999822 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:17:54.999826 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:17:54.999915 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:17:55.000208 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:17:55.000211 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:17:55.000213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:17:55.000442 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:17:55.000520 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0618 22:17:55.000935 (+ 415us) server_negotiation.cc:299] Negotiation successful
0618 22:17:55.001015 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260618 22:17:55.001329 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:54.991999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:46833 (local address 127.0.98.67:52347)
0618 22:17:54.993120 (+ 1121us) negotiation.cc:107] Waiting for socket to connect
0618 22:17:54.993131 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 22:17:54.993249 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:17:54.993833 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:17:54.993836 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:17:54.994070 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:17:54.994182 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:54.994189 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:54.995082 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:54.995085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:54.995641 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:54.995649 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:54.996355 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:54.996358 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:54.996432 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:54.997149 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:17:54.997161 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:17:54.997472 (+ 311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:17:54.999489 (+ 2017us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:17:54.999491 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:17:54.999493 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:17:54.999714 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:17:55.000024 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:17:55.000026 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:17:55.000027 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:17:55.000115 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:17:55.000667 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:17:55.000670 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:17:55.000764 (+ 94us) client_negotiation.cc:769] Sending connection context
0618 22:17:55.000890 (+ 126us) client_negotiation.cc:240] Negotiation successful
0618 22:17:55.001051 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1056,"thread_start_us":103,"threads_started":1}
Jun 18 22:17:55 dist-test-slave-g6r4 krb5kdc[2086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 18 22:17:55 dist-test-slave-g6r4 krb5kdc[2086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:17:55.021262 2180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:55.013548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38894 (local address 127.0.98.126:46833)
0618 22:17:55.013669 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0618 22:17:55.013673 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:17:55.013687 (+ 14us) server_negotiation.cc:407] Connection header received
0618 22:17:55.013728 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:17:55.013730 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:17:55.013764 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:17:55.013829 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:17:55.014590 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:55.015089 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:55.015884 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:55.016081 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:55.016152 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:55.018540 (+ 2388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:17:55.018543 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:17:55.018545 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:17:55.018565 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:17:55.019767 (+ 1202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:17:55.020116 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:17:55.020119 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:17:55.020120 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:17:55.020185 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:17:55.020537 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:17:55.020539 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:17:55.020540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:17:55.020691 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:17:55.020787 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0618 22:17:55.020994 (+ 207us) server_negotiation.cc:299] Negotiation successful
0618 22:17:55.021075 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260618 22:17:55.023857 2130 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38894:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 22:17:55.026736 2130 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:17:55.040340 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:55.034719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40906 (local address 127.0.98.66:46229)
0618 22:17:55.035150 (+ 431us) server_negotiation.cc:206] Beginning negotiation
0618 22:17:55.035165 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0618 22:17:55.035189 (+ 24us) server_negotiation.cc:407] Connection header received
0618 22:17:55.035389 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:17:55.035395 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:17:55.035723 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:17:55.035872 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:17:55.036213 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:55.037323 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:55.038940 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:55.039667 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:55.039788 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:55.039950 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0618 22:17:55.040073 (+ 123us) server_negotiation.cc:299] Negotiation successful
0618 22:17:55.040199 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":123,"threads_started":1}
I20260618 22:17:55.040331 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:55.034559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:46229 (local address 127.0.0.1:40906)
0618 22:17:55.035053 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0618 22:17:55.035098 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0618 22:17:55.035221 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:17:55.035880 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:17:55.035886 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:17:55.035904 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:17:55.036029 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:55.036036 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:55.037427 (+ 1391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:55.037433 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:55.038810 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:55.038820 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:55.039786 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:55.039789 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:55.039901 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:55.039944 (+ 43us) client_negotiation.cc:769] Sending connection context
0618 22:17:55.040048 (+ 104us) client_negotiation.cc:240] Negotiation successful
0618 22:17:55.040171 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"mutex_wait_us":62,"thread_start_us":80,"threads_started":1}
I20260618 22:17:55.041105 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:55.035597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39364 (local address 127.0.98.65:35281)
0618 22:17:55.036014 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0618 22:17:55.036031 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0618 22:17:55.036053 (+ 22us) server_negotiation.cc:407] Connection header received
0618 22:17:55.036129 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:17:55.036135 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:17:55.036321 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:17:55.036449 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:17:55.036731 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:55.037691 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:55.039372 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:55.040232 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:55.040353 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:55.040507 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0618 22:17:55.040882 (+ 375us) server_negotiation.cc:299] Negotiation successful
0618 22:17:55.040978 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":81,"threads_started":1}
I20260618 22:17:55.041177 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:55.035277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:33011 (local address 127.0.0.1:48362)
0618 22:17:55.036629 (+ 1352us) negotiation.cc:107] Waiting for socket to connect
0618 22:17:55.036640 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 22:17:55.036760 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:17:55.037190 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:17:55.037193 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:17:55.037211 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:17:55.037316 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:55.037322 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:55.038566 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:55.038569 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:55.039750 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:55.039760 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:55.040799 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:55.040802 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:55.040898 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:55.040918 (+ 20us) client_negotiation.cc:769] Sending connection context
0618 22:17:55.040982 (+ 64us) client_negotiation.cc:240] Negotiation successful
0618 22:17:55.041049 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1299,"mutex_wait_us":50,"thread_start_us":105,"threads_started":1}
I20260618 22:17:55.041230 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:55.035574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48362 (local address 127.0.98.67:33011)
0618 22:17:55.036284 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0618 22:17:55.036295 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0618 22:17:55.036738 (+ 443us) server_negotiation.cc:407] Connection header received
0618 22:17:55.036892 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:17:55.036897 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:17:55.037053 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:17:55.037189 (+ 136us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:17:55.037448 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:55.038440 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:55.039892 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:55.040675 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:55.040801 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:55.040951 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0618 22:17:55.041017 (+ 66us) server_negotiation.cc:299] Negotiation successful
0618 22:17:55.041098 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":102,"threads_started":1}
I20260618 22:17:55.041781 2632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:55.034749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:35281 (local address 127.0.0.1:39364)
0618 22:17:55.035659 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0618 22:17:55.035673 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:17:55.035754 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:17:55.036462 (+ 708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:17:55.036467 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:17:55.036487 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:17:55.036592 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:55.036598 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:55.037804 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:55.037807 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:55.039241 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:55.039250 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:55.040619 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:55.040623 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:55.040712 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:55.040742 (+ 30us) client_negotiation.cc:769] Sending connection context
0618 22:17:55.041594 (+ 852us) client_negotiation.cc:240] Negotiation successful
0618 22:17:55.041669 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":851,"mutex_wait_us":40,"thread_start_us":82,"threads_started":1}
I20260618 22:17:55.043597 2401 tablet_service.cc:1467] Processing CreateTablet for tablet ae86fe5b4486438680bf8aef5de578dc (DEFAULT_TABLE table=test-table [id=e3c661eac03c41b197429437e49b4630]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:17:55.043848 2548 tablet_service.cc:1467] Processing CreateTablet for tablet ae86fe5b4486438680bf8aef5de578dc (DEFAULT_TABLE table=test-table [id=e3c661eac03c41b197429437e49b4630]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:17:55.044752 2401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae86fe5b4486438680bf8aef5de578dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:17:55.044452 2268 tablet_service.cc:1467] Processing CreateTablet for tablet ae86fe5b4486438680bf8aef5de578dc (DEFAULT_TABLE table=test-table [id=e3c661eac03c41b197429437e49b4630]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:17:55.044847 2548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae86fe5b4486438680bf8aef5de578dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:17:55.045361 2268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae86fe5b4486438680bf8aef5de578dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:17:55.051308 2637 tablet_bootstrap.cc:492] T ae86fe5b4486438680bf8aef5de578dc P 24b2931f243c419a9d1c7d00c97defac: Bootstrap starting.
I20260618 22:17:55.052892 2638 tablet_bootstrap.cc:492] T ae86fe5b4486438680bf8aef5de578dc P 58a33666811f42fca694ee4b30acb12f: Bootstrap starting.
I20260618 22:17:55.053323 2639 tablet_bootstrap.cc:492] T ae86fe5b4486438680bf8aef5de578dc P c5de8b6fc77c4d1090acf81f46f74192: Bootstrap starting.
I20260618 22:17:55.053483 2637 tablet_bootstrap.cc:654] T ae86fe5b4486438680bf8aef5de578dc P 24b2931f243c419a9d1c7d00c97defac: Neither blocks nor log segments found. Creating new log.
I20260618 22:17:55.054243 2637 log.cc:824] T ae86fe5b4486438680bf8aef5de578dc P 24b2931f243c419a9d1c7d00c97defac: Log is configured to *not* fsync() on all Append() calls
I20260618 22:17:55.054863 2638 tablet_bootstrap.cc:654] T ae86fe5b4486438680bf8aef5de578dc P 58a33666811f42fca694ee4b30acb12f: Neither blocks nor log segments found. Creating new log.
I20260618 22:17:55.055231 2639 tablet_bootstrap.cc:654] T ae86fe5b4486438680bf8aef5de578dc P c5de8b6fc77c4d1090acf81f46f74192: Neither blocks nor log segments found. Creating new log.
I20260618 22:17:55.055800 2638 log.cc:824] T ae86fe5b4486438680bf8aef5de578dc P 58a33666811f42fca694ee4b30acb12f: Log is configured to *not* fsync() on all Append() calls
I20260618 22:17:55.056044 2639 log.cc:824] T ae86fe5b4486438680bf8aef5de578dc P c5de8b6fc77c4d1090acf81f46f74192: Log is configured to *not* fsync() on all Append() calls
I20260618 22:17:55.056083 2637 tablet_bootstrap.cc:492] T ae86fe5b4486438680bf8aef5de578dc P 24b2931f243c419a9d1c7d00c97defac: No bootstrap required, opened a new log
I20260618 22:17:55.056260 2637 ts_tablet_manager.cc:1397] T ae86fe5b4486438680bf8aef5de578dc P 24b2931f243c419a9d1c7d00c97defac: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260618 22:17:55.057745 2638 tablet_bootstrap.cc:492] T ae86fe5b4486438680bf8aef5de578dc P 58a33666811f42fca694ee4b30acb12f: No bootstrap required, opened a new log
I20260618 22:17:55.057932 2638 ts_tablet_manager.cc:1397] T ae86fe5b4486438680bf8aef5de578dc P 58a33666811f42fca694ee4b30acb12f: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260618 22:17:55.057977 2639 tablet_bootstrap.cc:492] T ae86fe5b4486438680bf8aef5de578dc P c5de8b6fc77c4d1090acf81f46f74192: No bootstrap required, opened a new log
I20260618 22:17:55.058135 2639 ts_tablet_manager.cc:1397] T ae86fe5b4486438680bf8aef5de578dc P c5de8b6fc77c4d1090acf81f46f74192: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260618 22:17:55.059473 2637 raft_consensus.cc:348] T ae86fe5b4486438680bf8aef5de578dc P 24b2931f243c419a9d1c7d00c97defac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24b2931f243c419a9d1c7d00c97defac" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46229 } } peers { permanent_uuid: "c5de8b6fc77c4d1090acf81f46f74192" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35281 } } peers { permanent_uuid: "58a33666811f42fca694ee4b30acb12f" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33011 } }
I20260618 22:17:55.059700 2637 raft_consensus.cc:374] T ae86fe5b4486438680bf8aef5de578dc P 24b2931f243c419a9d1c7d00c97defac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:17:55.059769 2637 raft_consensus.cc:729] T ae86fe5b4486438680bf8aef5de578dc P 24b2931f243c419a9d1c7d00c97defac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24b2931f243c419a9d1c7d00c97defac, State: Initialized, Role: FOLLOWER
I20260618 22:17:55.060211 2637 consensus_queue.cc:260] T ae86fe5b4486438680bf8aef5de578dc P 24b2931f243c419a9d1c7d00c97defac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24b2931f243c419a9d1c7d00c97defac" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46229 } } peers { permanent_uuid: "c5de8b6fc77c4d1090acf81f46f74192" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35281 } } peers { permanent_uuid: "58a33666811f42fca694ee4b30acb12f" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33011 } }
I20260618 22:17:55.061131 2637 ts_tablet_manager.cc:1428] T ae86fe5b4486438680bf8aef5de578dc P 24b2931f243c419a9d1c7d00c97defac: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260618 22:17:55.061475 2474 heartbeater.cc:503] Master 127.0.98.126:46833 was elected leader, sending a full tablet report...
I20260618 22:17:55.061678 2638 raft_consensus.cc:348] T ae86fe5b4486438680bf8aef5de578dc P 58a33666811f42fca694ee4b30acb12f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24b2931f243c419a9d1c7d00c97defac" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46229 } } peers { permanent_uuid: "c5de8b6fc77c4d1090acf81f46f74192" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35281 } } peers { permanent_uuid: "58a33666811f42fca694ee4b30acb12f" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33011 } }
I20260618 22:17:55.061678 2639 raft_consensus.cc:348] T ae86fe5b4486438680bf8aef5de578dc P c5de8b6fc77c4d1090acf81f46f74192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24b2931f243c419a9d1c7d00c97defac" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46229 } } peers { permanent_uuid: "c5de8b6fc77c4d1090acf81f46f74192" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35281 } } peers { permanent_uuid: "58a33666811f42fca694ee4b30acb12f" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33011 } }
I20260618 22:17:55.061915 2638 raft_consensus.cc:374] T ae86fe5b4486438680bf8aef5de578dc P 58a33666811f42fca694ee4b30acb12f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:17:55.061926 2639 raft_consensus.cc:374] T ae86fe5b4486438680bf8aef5de578dc P c5de8b6fc77c4d1090acf81f46f74192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:17:55.061970 2638 raft_consensus.cc:729] T ae86fe5b4486438680bf8aef5de578dc P 58a33666811f42fca694ee4b30acb12f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58a33666811f42fca694ee4b30acb12f, State: Initialized, Role: FOLLOWER
I20260618 22:17:55.061970 2639 raft_consensus.cc:729] T ae86fe5b4486438680bf8aef5de578dc P c5de8b6fc77c4d1090acf81f46f74192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5de8b6fc77c4d1090acf81f46f74192, State: Initialized, Role: FOLLOWER
I20260618 22:17:55.062382 2638 consensus_queue.cc:260] T ae86fe5b4486438680bf8aef5de578dc P 58a33666811f42fca694ee4b30acb12f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24b2931f243c419a9d1c7d00c97defac" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46229 } } peers { permanent_uuid: "c5de8b6fc77c4d1090acf81f46f74192" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35281 } } peers { permanent_uuid: "58a33666811f42fca694ee4b30acb12f" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33011 } }
I20260618 22:17:55.062382 2639 consensus_queue.cc:260] T ae86fe5b4486438680bf8aef5de578dc P c5de8b6fc77c4d1090acf81f46f74192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24b2931f243c419a9d1c7d00c97defac" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46229 } } peers { permanent_uuid: "c5de8b6fc77c4d1090acf81f46f74192" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35281 } } peers { permanent_uuid: "58a33666811f42fca694ee4b30acb12f" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33011 } }
I20260618 22:17:55.062994 2334 heartbeater.cc:503] Master 127.0.98.126:46833 was elected leader, sending a full tablet report...
I20260618 22:17:55.063282 2638 ts_tablet_manager.cc:1428] T ae86fe5b4486438680bf8aef5de578dc P 58a33666811f42fca694ee4b30acb12f: Time spent starting tablet: real 0.005s user 0.002s sys 0.003s
I20260618 22:17:55.063282 2639 ts_tablet_manager.cc:1428] T ae86fe5b4486438680bf8aef5de578dc P c5de8b6fc77c4d1090acf81f46f74192: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260618 22:17:55.063578 2614 heartbeater.cc:503] Master 127.0.98.126:46833 was elected leader, sending a full tablet report...
I20260618 22:17:55.092087 2643 raft_consensus.cc:482] T ae86fe5b4486438680bf8aef5de578dc P 24b2931f243c419a9d1c7d00c97defac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:17:55.092257 2643 raft_consensus.cc:504] T ae86fe5b4486438680bf8aef5de578dc P 24b2931f243c419a9d1c7d00c97defac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24b2931f243c419a9d1c7d00c97defac" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46229 } } peers { permanent_uuid: "c5de8b6fc77c4d1090acf81f46f74192" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35281 } } peers { permanent_uuid: "58a33666811f42fca694ee4b30acb12f" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33011 } }
I20260618 22:17:55.093358 2643 leader_election.cc:288] T ae86fe5b4486438680bf8aef5de578dc P 24b2931f243c419a9d1c7d00c97defac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5de8b6fc77c4d1090acf81f46f74192 (127.0.98.65:35281), 58a33666811f42fca694ee4b30acb12f (127.0.98.67:33011)
I20260618 22:17:55.097363 2367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:55.093532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:35281 (local address 127.0.98.66:54723)
0618 22:17:55.093712 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0618 22:17:55.093728 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 22:17:55.093815 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:17:55.094170 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:17:55.094172 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:17:55.094184 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:17:55.094319 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:55.094326 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:55.095124 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:55.095126 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:55.096257 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:55.096267 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:55.096977 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:55.096979 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:55.097072 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:55.097091 (+ 19us) client_negotiation.cc:769] Sending connection context
0618 22:17:55.097156 (+ 65us) client_negotiation.cc:240] Negotiation successful
0618 22:17:55.097209 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103}
I20260618 22:17:55.097399 2636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:55.093600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:54723 (local address 127.0.98.65:35281)
0618 22:17:55.093750 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0618 22:17:55.093756 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:17:55.093772 (+ 16us) server_negotiation.cc:407] Connection header received
0618 22:17:55.093984 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:17:55.093987 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:17:55.094053 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:17:55.094137 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:17:55.094431 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:55.095024 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:55.096373 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:55.096873 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:55.096971 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:55.097032 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0618 22:17:55.097183 (+ 151us) server_negotiation.cc:299] Negotiation successful
0618 22:17:55.097260 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260618 22:17:55.097792 2635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:55.093745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:35075 (local address 127.0.98.67:33011)
0618 22:17:55.093898 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0618 22:17:55.093903 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:17:55.094054 (+ 151us) server_negotiation.cc:407] Connection header received
0618 22:17:55.094126 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:17:55.094130 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:17:55.094173 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:17:55.094255 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:17:55.094837 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:55.095416 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:55.096651 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:55.097201 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:55.097307 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:55.097359 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0618 22:17:55.097586 (+ 227us) server_negotiation.cc:299] Negotiation successful
0618 22:17:55.097664 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260618 22:17:55.097997 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae86fe5b4486438680bf8aef5de578dc" candidate_uuid: "24b2931f243c419a9d1c7d00c97defac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5de8b6fc77c4d1090acf81f46f74192" is_pre_election: true
I20260618 22:17:55.098176 2646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:55.093590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:33011 (local address 127.0.98.66:35075)
0618 22:17:55.093975 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0618 22:17:55.093985 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0618 22:17:55.094056 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:17:55.094264 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:17:55.094266 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:17:55.094280 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:17:55.094733 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:55.094740 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:55.095525 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:55.095528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:55.096554 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:55.096561 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:55.097329 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:55.097332 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:55.097433 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:55.097456 (+ 23us) client_negotiation.cc:769] Sending connection context
0618 22:17:55.097988 (+ 532us) client_negotiation.cc:240] Negotiation successful
0618 22:17:55.098058 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":121,"threads_started":1}
I20260618 22:17:55.098320 2288 raft_consensus.cc:2454] T ae86fe5b4486438680bf8aef5de578dc P c5de8b6fc77c4d1090acf81f46f74192 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24b2931f243c419a9d1c7d00c97defac in term 0.
I20260618 22:17:55.098824 2360 leader_election.cc:302] T ae86fe5b4486438680bf8aef5de578dc P 24b2931f243c419a9d1c7d00c97defac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24b2931f243c419a9d1c7d00c97defac, c5de8b6fc77c4d1090acf81f46f74192; no voters:
I20260618 22:17:55.098830 2568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae86fe5b4486438680bf8aef5de578dc" candidate_uuid: "24b2931f243c419a9d1c7d00c97defac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58a33666811f42fca694ee4b30acb12f" is_pre_election: true
I20260618 22:17:55.099086 2568 raft_consensus.cc:2454] T ae86fe5b4486438680bf8aef5de578dc P 58a33666811f42fca694ee4b30acb12f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24b2931f243c419a9d1c7d00c97defac in term 0.
I20260618 22:17:55.099097 2643 raft_consensus.cc:2781] T ae86fe5b4486438680bf8aef5de578dc P 24b2931f243c419a9d1c7d00c97defac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:17:55.099184 2643 raft_consensus.cc:482] T ae86fe5b4486438680bf8aef5de578dc P 24b2931f243c419a9d1c7d00c97defac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:17:55.099264 2643 raft_consensus.cc:3037] T ae86fe5b4486438680bf8aef5de578dc P 24b2931f243c419a9d1c7d00c97defac [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:17:55.100281 2643 raft_consensus.cc:504] T ae86fe5b4486438680bf8aef5de578dc P 24b2931f243c419a9d1c7d00c97defac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24b2931f243c419a9d1c7d00c97defac" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46229 } } peers { permanent_uuid: "c5de8b6fc77c4d1090acf81f46f74192" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35281 } } peers { permanent_uuid: "58a33666811f42fca694ee4b30acb12f" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33011 } }
I20260618 22:17:55.100649 2643 leader_election.cc:288] T ae86fe5b4486438680bf8aef5de578dc P 24b2931f243c419a9d1c7d00c97defac [CANDIDATE]: Term 1 election: Requested vote from peers c5de8b6fc77c4d1090acf81f46f74192 (127.0.98.65:35281), 58a33666811f42fca694ee4b30acb12f (127.0.98.67:33011)
I20260618 22:17:55.100901 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae86fe5b4486438680bf8aef5de578dc" candidate_uuid: "24b2931f243c419a9d1c7d00c97defac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5de8b6fc77c4d1090acf81f46f74192"
I20260618 22:17:55.100981 2568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae86fe5b4486438680bf8aef5de578dc" candidate_uuid: "24b2931f243c419a9d1c7d00c97defac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58a33666811f42fca694ee4b30acb12f"
I20260618 22:17:55.101090 2568 raft_consensus.cc:3037] T ae86fe5b4486438680bf8aef5de578dc P 58a33666811f42fca694ee4b30acb12f [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:17:55.101094 2288 raft_consensus.cc:3037] T ae86fe5b4486438680bf8aef5de578dc P c5de8b6fc77c4d1090acf81f46f74192 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:17:55.102114 2288 raft_consensus.cc:2454] T ae86fe5b4486438680bf8aef5de578dc P c5de8b6fc77c4d1090acf81f46f74192 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24b2931f243c419a9d1c7d00c97defac in term 1.
I20260618 22:17:55.102183 2568 raft_consensus.cc:2454] T ae86fe5b4486438680bf8aef5de578dc P 58a33666811f42fca694ee4b30acb12f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24b2931f243c419a9d1c7d00c97defac in term 1.
I20260618 22:17:55.102451 2360 leader_election.cc:302] T ae86fe5b4486438680bf8aef5de578dc P 24b2931f243c419a9d1c7d00c97defac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24b2931f243c419a9d1c7d00c97defac, c5de8b6fc77c4d1090acf81f46f74192; no voters:
I20260618 22:17:55.102649 2643 raft_consensus.cc:2781] T ae86fe5b4486438680bf8aef5de578dc P 24b2931f243c419a9d1c7d00c97defac [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:17:55.102867 2643 raft_consensus.cc:686] T ae86fe5b4486438680bf8aef5de578dc P 24b2931f243c419a9d1c7d00c97defac [term 1 LEADER]: Becoming Leader. State: Replica: 24b2931f243c419a9d1c7d00c97defac, State: Running, Role: LEADER
I20260618 22:17:55.103168 2643 consensus_queue.cc:237] T ae86fe5b4486438680bf8aef5de578dc P 24b2931f243c419a9d1c7d00c97defac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24b2931f243c419a9d1c7d00c97defac" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46229 } } peers { permanent_uuid: "c5de8b6fc77c4d1090acf81f46f74192" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35281 } } peers { permanent_uuid: "58a33666811f42fca694ee4b30acb12f" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33011 } }
I20260618 22:17:55.106137 2130 catalog_manager.cc:5495] T ae86fe5b4486438680bf8aef5de578dc P 24b2931f243c419a9d1c7d00c97defac reported cstate change: term changed from 0 to 1, leader changed from <none> to 24b2931f243c419a9d1c7d00c97defac (127.0.98.66). New cstate: current_term: 1 leader_uuid: "24b2931f243c419a9d1c7d00c97defac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24b2931f243c419a9d1c7d00c97defac" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46229 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c5de8b6fc77c4d1090acf81f46f74192" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58a33666811f42fca694ee4b30acb12f" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33011 } health_report { overall_health: UNKNOWN } } }
Jun 18 22:17:55 dist-test-slave-g6r4 krb5kdc[2086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821074, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:17:55.139060 2654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:55.131320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:46833 (local address 127.0.0.1:38902)
0618 22:17:55.131575 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0618 22:17:55.131588 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:17:55.131678 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:17:55.131888 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:17:55.131890 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:17:55.132234 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:17:55.132321 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:55.132326 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:55.133040 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:55.133042 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:55.133583 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:55.133591 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:55.134030 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:55.134032 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:55.134094 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:55.134608 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:17:55.134619 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:17:55.136017 (+ 1398us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:17:55.137722 (+ 1705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:17:55.137729 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:17:55.137734 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:17:55.138001 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:17:55.138232 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:17:55.138234 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:17:55.138235 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:17:55.138310 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:17:55.138589 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:17:55.138594 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:17:55.138680 (+ 86us) client_negotiation.cc:769] Sending connection context
0618 22:17:55.138808 (+ 128us) client_negotiation.cc:240] Negotiation successful
0618 22:17:55.138916 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":76,"threads_started":1}
I20260618 22:17:55.139078 2180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:55.131442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38902 (local address 127.0.98.126:46833)
0618 22:17:55.131602 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0618 22:17:55.131606 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:17:55.131653 (+ 47us) server_negotiation.cc:407] Connection header received
0618 22:17:55.131749 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:17:55.131751 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:17:55.131786 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:17:55.131853 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:17:55.132414 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:55.132940 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:55.133701 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:55.133933 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:55.134032 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:55.136201 (+ 2169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:17:55.136204 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:17:55.136206 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:17:55.136225 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:17:55.137636 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:17:55.138071 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:17:55.138074 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:17:55.138075 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:17:55.138156 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:17:55.138365 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:17:55.138367 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:17:55.138368 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:17:55.138514 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:17:55.138578 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0618 22:17:55.138862 (+ 284us) server_negotiation.cc:299] Negotiation successful
0618 22:17:55.138966 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":16,"server-negotiator.queue_time_us":78}
I20260618 22:17:55.142668 2130 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:38902:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260618 22:17:55.143301 2130 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:17:55.147742 2401 tablet_service.cc:1467] Processing CreateTablet for tablet 228e0bfc4755408f936b4539fb25ea71 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=359ae883df2a46749ed00f5b79a59b5a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260618 22:17:55.147665 2268 tablet_service.cc:1467] Processing CreateTablet for tablet 228e0bfc4755408f936b4539fb25ea71 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=359ae883df2a46749ed00f5b79a59b5a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260618 22:17:55.147979 2548 tablet_service.cc:1467] Processing CreateTablet for tablet 228e0bfc4755408f936b4539fb25ea71 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=359ae883df2a46749ed00f5b79a59b5a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260618 22:17:55.148371 2401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 228e0bfc4755408f936b4539fb25ea71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:17:55.148375 2268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 228e0bfc4755408f936b4539fb25ea71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:17:55.148491 2548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 228e0bfc4755408f936b4539fb25ea71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:17:55.150674 2637 tablet_bootstrap.cc:492] T 228e0bfc4755408f936b4539fb25ea71 P 24b2931f243c419a9d1c7d00c97defac: Bootstrap starting.
I20260618 22:17:55.151124 2639 tablet_bootstrap.cc:492] T 228e0bfc4755408f936b4539fb25ea71 P c5de8b6fc77c4d1090acf81f46f74192: Bootstrap starting.
I20260618 22:17:55.151124 2638 tablet_bootstrap.cc:492] T 228e0bfc4755408f936b4539fb25ea71 P 58a33666811f42fca694ee4b30acb12f: Bootstrap starting.
I20260618 22:17:55.151624 2637 tablet_bootstrap.cc:654] T 228e0bfc4755408f936b4539fb25ea71 P 24b2931f243c419a9d1c7d00c97defac: Neither blocks nor log segments found. Creating new log.
I20260618 22:17:55.152017 2639 tablet_bootstrap.cc:654] T 228e0bfc4755408f936b4539fb25ea71 P c5de8b6fc77c4d1090acf81f46f74192: Neither blocks nor log segments found. Creating new log.
I20260618 22:17:55.152017 2638 tablet_bootstrap.cc:654] T 228e0bfc4755408f936b4539fb25ea71 P 58a33666811f42fca694ee4b30acb12f: Neither blocks nor log segments found. Creating new log.
I20260618 22:17:55.152962 2637 tablet_bootstrap.cc:492] T 228e0bfc4755408f936b4539fb25ea71 P 24b2931f243c419a9d1c7d00c97defac: No bootstrap required, opened a new log
I20260618 22:17:55.153043 2637 ts_tablet_manager.cc:1397] T 228e0bfc4755408f936b4539fb25ea71 P 24b2931f243c419a9d1c7d00c97defac: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260618 22:17:55.153185 2638 tablet_bootstrap.cc:492] T 228e0bfc4755408f936b4539fb25ea71 P 58a33666811f42fca694ee4b30acb12f: No bootstrap required, opened a new log
I20260618 22:17:55.153256 2638 ts_tablet_manager.cc:1397] T 228e0bfc4755408f936b4539fb25ea71 P 58a33666811f42fca694ee4b30acb12f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260618 22:17:55.153412 2639 tablet_bootstrap.cc:492] T 228e0bfc4755408f936b4539fb25ea71 P c5de8b6fc77c4d1090acf81f46f74192: No bootstrap required, opened a new log
I20260618 22:17:55.153492 2639 ts_tablet_manager.cc:1397] T 228e0bfc4755408f936b4539fb25ea71 P c5de8b6fc77c4d1090acf81f46f74192: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260618 22:17:55.153488 2637 raft_consensus.cc:348] T 228e0bfc4755408f936b4539fb25ea71 P 24b2931f243c419a9d1c7d00c97defac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5de8b6fc77c4d1090acf81f46f74192" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35281 } } peers { permanent_uuid: "24b2931f243c419a9d1c7d00c97defac" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46229 } } peers { permanent_uuid: "58a33666811f42fca694ee4b30acb12f" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33011 } }
I20260618 22:17:55.153585 2637 raft_consensus.cc:374] T 228e0bfc4755408f936b4539fb25ea71 P 24b2931f243c419a9d1c7d00c97defac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:17:55.153620 2637 raft_consensus.cc:729] T 228e0bfc4755408f936b4539fb25ea71 P 24b2931f243c419a9d1c7d00c97defac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24b2931f243c419a9d1c7d00c97defac, State: Initialized, Role: FOLLOWER
I20260618 22:17:55.153663 2638 raft_consensus.cc:348] T 228e0bfc4755408f936b4539fb25ea71 P 58a33666811f42fca694ee4b30acb12f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5de8b6fc77c4d1090acf81f46f74192" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35281 } } peers { permanent_uuid: "24b2931f243c419a9d1c7d00c97defac" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46229 } } peers { permanent_uuid: "58a33666811f42fca694ee4b30acb12f" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33011 } }
I20260618 22:17:55.153750 2638 raft_consensus.cc:374] T 228e0bfc4755408f936b4539fb25ea71 P 58a33666811f42fca694ee4b30acb12f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:17:55.153781 2638 raft_consensus.cc:729] T 228e0bfc4755408f936b4539fb25ea71 P 58a33666811f42fca694ee4b30acb12f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58a33666811f42fca694ee4b30acb12f, State: Initialized, Role: FOLLOWER
I20260618 22:17:55.153748 2637 consensus_queue.cc:260] T 228e0bfc4755408f936b4539fb25ea71 P 24b2931f243c419a9d1c7d00c97defac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5de8b6fc77c4d1090acf81f46f74192" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35281 } } peers { permanent_uuid: "24b2931f243c419a9d1c7d00c97defac" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46229 } } peers { permanent_uuid: "58a33666811f42fca694ee4b30acb12f" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33011 } }
I20260618 22:17:55.153913 2638 consensus_queue.cc:260] T 228e0bfc4755408f936b4539fb25ea71 P 58a33666811f42fca694ee4b30acb12f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5de8b6fc77c4d1090acf81f46f74192" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35281 } } peers { permanent_uuid: "24b2931f243c419a9d1c7d00c97defac" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46229 } } peers { permanent_uuid: "58a33666811f42fca694ee4b30acb12f" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33011 } }
I20260618 22:17:55.154037 2637 ts_tablet_manager.cc:1428] T 228e0bfc4755408f936b4539fb25ea71 P 24b2931f243c419a9d1c7d00c97defac: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260618 22:17:55.153990 2639 raft_consensus.cc:348] T 228e0bfc4755408f936b4539fb25ea71 P c5de8b6fc77c4d1090acf81f46f74192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5de8b6fc77c4d1090acf81f46f74192" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35281 } } peers { permanent_uuid: "24b2931f243c419a9d1c7d00c97defac" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46229 } } peers { permanent_uuid: "58a33666811f42fca694ee4b30acb12f" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33011 } }
I20260618 22:17:55.154105 2639 raft_consensus.cc:374] T 228e0bfc4755408f936b4539fb25ea71 P c5de8b6fc77c4d1090acf81f46f74192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:17:55.154140 2639 raft_consensus.cc:729] T 228e0bfc4755408f936b4539fb25ea71 P c5de8b6fc77c4d1090acf81f46f74192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5de8b6fc77c4d1090acf81f46f74192, State: Initialized, Role: FOLLOWER
I20260618 22:17:55.154177 2645 tablet_replica.cc:439] T 228e0bfc4755408f936b4539fb25ea71 P 58a33666811f42fca694ee4b30acb12f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5de8b6fc77c4d1090acf81f46f74192" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35281 } } peers { permanent_uuid: "24b2931f243c419a9d1c7d00c97defac" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46229 } } peers { permanent_uuid: "58a33666811f42fca694ee4b30acb12f" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33011 } } }
I20260618 22:17:55.154256 2639 consensus_queue.cc:260] T 228e0bfc4755408f936b4539fb25ea71 P c5de8b6fc77c4d1090acf81f46f74192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5de8b6fc77c4d1090acf81f46f74192" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35281 } } peers { permanent_uuid: "24b2931f243c419a9d1c7d00c97defac" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46229 } } peers { permanent_uuid: "58a33666811f42fca694ee4b30acb12f" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33011 } }
I20260618 22:17:55.154330 2645 tablet_replica.cc:442] T 228e0bfc4755408f936b4539fb25ea71 P 58a33666811f42fca694ee4b30acb12f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 22:17:55.154500 2639 ts_tablet_manager.cc:1428] T 228e0bfc4755408f936b4539fb25ea71 P c5de8b6fc77c4d1090acf81f46f74192: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260618 22:17:55.154595 2643 tablet_replica.cc:439] T 228e0bfc4755408f936b4539fb25ea71 P 24b2931f243c419a9d1c7d00c97defac: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5de8b6fc77c4d1090acf81f46f74192" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35281 } } peers { permanent_uuid: "24b2931f243c419a9d1c7d00c97defac" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46229 } } peers { permanent_uuid: "58a33666811f42fca694ee4b30acb12f" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33011 } } }
I20260618 22:17:55.154721 2643 tablet_replica.cc:442] T 228e0bfc4755408f936b4539fb25ea71 P 24b2931f243c419a9d1c7d00c97defac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 22:17:55.155068 2644 tablet_replica.cc:439] T 228e0bfc4755408f936b4539fb25ea71 P c5de8b6fc77c4d1090acf81f46f74192: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5de8b6fc77c4d1090acf81f46f74192" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35281 } } peers { permanent_uuid: "24b2931f243c419a9d1c7d00c97defac" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46229 } } peers { permanent_uuid: "58a33666811f42fca694ee4b30acb12f" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33011 } } }
I20260618 22:17:55.155193 2644 tablet_replica.cc:442] T 228e0bfc4755408f936b4539fb25ea71 P c5de8b6fc77c4d1090acf81f46f74192: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 22:17:55.154163 2638 ts_tablet_manager.cc:1428] T 228e0bfc4755408f936b4539fb25ea71 P 58a33666811f42fca694ee4b30acb12f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
W20260618 22:17:55.237382 2615 tablet.cc:2369] T 228e0bfc4755408f936b4539fb25ea71 P 58a33666811f42fca694ee4b30acb12f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260618 22:17:55.264765 2475 tablet.cc:2369] T 228e0bfc4755408f936b4539fb25ea71 P 24b2931f243c419a9d1c7d00c97defac: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260618 22:17:55.312759 2335 tablet.cc:2369] T 228e0bfc4755408f936b4539fb25ea71 P c5de8b6fc77c4d1090acf81f46f74192: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:17:55.423866 2643 raft_consensus.cc:482] T 228e0bfc4755408f936b4539fb25ea71 P 24b2931f243c419a9d1c7d00c97defac [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:17:55.424006 2643 raft_consensus.cc:504] T 228e0bfc4755408f936b4539fb25ea71 P 24b2931f243c419a9d1c7d00c97defac [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5de8b6fc77c4d1090acf81f46f74192" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35281 } } peers { permanent_uuid: "24b2931f243c419a9d1c7d00c97defac" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46229 } } peers { permanent_uuid: "58a33666811f42fca694ee4b30acb12f" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33011 } }
I20260618 22:17:55.424479 2643 leader_election.cc:288] T 228e0bfc4755408f936b4539fb25ea71 P 24b2931f243c419a9d1c7d00c97defac [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5de8b6fc77c4d1090acf81f46f74192 (127.0.98.65:35281), 58a33666811f42fca694ee4b30acb12f (127.0.98.67:33011)
I20260618 22:17:55.425006 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228e0bfc4755408f936b4539fb25ea71" candidate_uuid: "24b2931f243c419a9d1c7d00c97defac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5de8b6fc77c4d1090acf81f46f74192" is_pre_election: true
I20260618 22:17:55.425006 2568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228e0bfc4755408f936b4539fb25ea71" candidate_uuid: "24b2931f243c419a9d1c7d00c97defac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58a33666811f42fca694ee4b30acb12f" is_pre_election: true
I20260618 22:17:55.425158 2288 raft_consensus.cc:2454] T 228e0bfc4755408f936b4539fb25ea71 P c5de8b6fc77c4d1090acf81f46f74192 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24b2931f243c419a9d1c7d00c97defac in term 0.
I20260618 22:17:55.425158 2568 raft_consensus.cc:2454] T 228e0bfc4755408f936b4539fb25ea71 P 58a33666811f42fca694ee4b30acb12f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24b2931f243c419a9d1c7d00c97defac in term 0.
I20260618 22:17:55.425486 2362 leader_election.cc:302] T 228e0bfc4755408f936b4539fb25ea71 P 24b2931f243c419a9d1c7d00c97defac [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24b2931f243c419a9d1c7d00c97defac, 58a33666811f42fca694ee4b30acb12f; no voters:
I20260618 22:17:55.425689 2643 raft_consensus.cc:2781] T 228e0bfc4755408f936b4539fb25ea71 P 24b2931f243c419a9d1c7d00c97defac [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:17:55.425755 2643 raft_consensus.cc:482] T 228e0bfc4755408f936b4539fb25ea71 P 24b2931f243c419a9d1c7d00c97defac [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:17:55.425786 2643 raft_consensus.cc:3037] T 228e0bfc4755408f936b4539fb25ea71 P 24b2931f243c419a9d1c7d00c97defac [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:17:55.426724 2643 raft_consensus.cc:504] T 228e0bfc4755408f936b4539fb25ea71 P 24b2931f243c419a9d1c7d00c97defac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5de8b6fc77c4d1090acf81f46f74192" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35281 } } peers { permanent_uuid: "24b2931f243c419a9d1c7d00c97defac" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46229 } } peers { permanent_uuid: "58a33666811f42fca694ee4b30acb12f" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33011 } }
I20260618 22:17:55.427016 2643 leader_election.cc:288] T 228e0bfc4755408f936b4539fb25ea71 P 24b2931f243c419a9d1c7d00c97defac [CANDIDATE]: Term 1 election: Requested vote from peers c5de8b6fc77c4d1090acf81f46f74192 (127.0.98.65:35281), 58a33666811f42fca694ee4b30acb12f (127.0.98.67:33011)
I20260618 22:17:55.427345 2568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228e0bfc4755408f936b4539fb25ea71" candidate_uuid: "24b2931f243c419a9d1c7d00c97defac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58a33666811f42fca694ee4b30acb12f"
I20260618 22:17:55.427354 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "228e0bfc4755408f936b4539fb25ea71" candidate_uuid: "24b2931f243c419a9d1c7d00c97defac" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5de8b6fc77c4d1090acf81f46f74192"
I20260618 22:17:55.427477 2288 raft_consensus.cc:3037] T 228e0bfc4755408f936b4539fb25ea71 P c5de8b6fc77c4d1090acf81f46f74192 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:17:55.427474 2568 raft_consensus.cc:3037] T 228e0bfc4755408f936b4539fb25ea71 P 58a33666811f42fca694ee4b30acb12f [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:17:55.428367 2288 raft_consensus.cc:2454] T 228e0bfc4755408f936b4539fb25ea71 P c5de8b6fc77c4d1090acf81f46f74192 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24b2931f243c419a9d1c7d00c97defac in term 1.
I20260618 22:17:55.428382 2568 raft_consensus.cc:2454] T 228e0bfc4755408f936b4539fb25ea71 P 58a33666811f42fca694ee4b30acb12f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24b2931f243c419a9d1c7d00c97defac in term 1.
I20260618 22:17:55.428628 2360 leader_election.cc:302] T 228e0bfc4755408f936b4539fb25ea71 P 24b2931f243c419a9d1c7d00c97defac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24b2931f243c419a9d1c7d00c97defac, c5de8b6fc77c4d1090acf81f46f74192; no voters:
I20260618 22:17:55.428800 2643 raft_consensus.cc:2781] T 228e0bfc4755408f936b4539fb25ea71 P 24b2931f243c419a9d1c7d00c97defac [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:17:55.428938 2643 raft_consensus.cc:686] T 228e0bfc4755408f936b4539fb25ea71 P 24b2931f243c419a9d1c7d00c97defac [term 1 LEADER]: Becoming Leader. State: Replica: 24b2931f243c419a9d1c7d00c97defac, State: Running, Role: LEADER
I20260618 22:17:55.429101 2643 consensus_queue.cc:237] T 228e0bfc4755408f936b4539fb25ea71 P 24b2931f243c419a9d1c7d00c97defac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5de8b6fc77c4d1090acf81f46f74192" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35281 } } peers { permanent_uuid: "24b2931f243c419a9d1c7d00c97defac" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46229 } } peers { permanent_uuid: "58a33666811f42fca694ee4b30acb12f" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33011 } }
I20260618 22:17:55.429781 2647 tablet_replica.cc:439] T 228e0bfc4755408f936b4539fb25ea71 P 24b2931f243c419a9d1c7d00c97defac: TxnStatusTablet state changed. Reason: New leader 24b2931f243c419a9d1c7d00c97defac. Latest consensus state: current_term: 1 leader_uuid: "24b2931f243c419a9d1c7d00c97defac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5de8b6fc77c4d1090acf81f46f74192" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35281 } } peers { permanent_uuid: "24b2931f243c419a9d1c7d00c97defac" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46229 } } peers { permanent_uuid: "58a33666811f42fca694ee4b30acb12f" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33011 } } }
I20260618 22:17:55.429891 2647 tablet_replica.cc:442] T 228e0bfc4755408f936b4539fb25ea71 P 24b2931f243c419a9d1c7d00c97defac: This TxnStatusTablet replica's current role is: LEADER
I20260618 22:17:55.430184 2659 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260618 22:17:55.430444 2659 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260618 22:17:55.430902 2129 catalog_manager.cc:5495] T 228e0bfc4755408f936b4539fb25ea71 P 24b2931f243c419a9d1c7d00c97defac reported cstate change: term changed from 0 to 1, leader changed from <none> to 24b2931f243c419a9d1c7d00c97defac (127.0.98.66). New cstate: current_term: 1 leader_uuid: "24b2931f243c419a9d1c7d00c97defac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5de8b6fc77c4d1090acf81f46f74192" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24b2931f243c419a9d1c7d00c97defac" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46229 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "58a33666811f42fca694ee4b30acb12f" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33011 } health_report { overall_health: UNKNOWN } } }
I20260618 22:17:55.447237 2654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:55.443216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:46229 (local address 127.0.0.1:40922)
0618 22:17:55.443374 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0618 22:17:55.443391 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0618 22:17:55.443484 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:17:55.444010 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:17:55.444012 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:17:55.444020 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0618 22:17:55.444105 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:55.444109 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:55.445082 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:55.445084 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:55.445662 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:55.445669 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:55.446211 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:55.446215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:55.446300 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:55.446341 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0618 22:17:55.447010 (+ 669us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0618 22:17:55.447020 (+ 10us) client_negotiation.cc:769] Sending connection context
0618 22:17:55.447069 (+ 49us) client_negotiation.cc:240] Negotiation successful
0618 22:17:55.447135 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260618 22:17:55.447257 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:55.443630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40922 (local address 127.0.98.66:46229)
0618 22:17:55.443749 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0618 22:17:55.443754 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:17:55.443767 (+ 13us) server_negotiation.cc:407] Connection header received
0618 22:17:55.443830 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:17:55.443833 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:17:55.443889 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:17:55.443990 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:17:55.444202 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:55.444990 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:55.445763 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:55.446109 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:55.446208 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:55.446468 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:17:55.446919 (+ 451us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:17:55.447028 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0618 22:17:55.447082 (+ 54us) server_negotiation.cc:299] Negotiation successful
0618 22:17:55.447134 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260618 22:17:55.451717 2568 raft_consensus.cc:1261] T 228e0bfc4755408f936b4539fb25ea71 P 58a33666811f42fca694ee4b30acb12f [term 1 FOLLOWER]: Refusing update from remote peer 24b2931f243c419a9d1c7d00c97defac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:17:55.451772 2288 raft_consensus.cc:1261] T 228e0bfc4755408f936b4539fb25ea71 P c5de8b6fc77c4d1090acf81f46f74192 [term 1 FOLLOWER]: Refusing update from remote peer 24b2931f243c419a9d1c7d00c97defac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:17:55.452389 2647 consensus_queue.cc:1035] T 228e0bfc4755408f936b4539fb25ea71 P 24b2931f243c419a9d1c7d00c97defac [LEADER]: Connected to new peer: Peer: permanent_uuid: "58a33666811f42fca694ee4b30acb12f" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:17:55.452548 2643 consensus_queue.cc:1035] T 228e0bfc4755408f936b4539fb25ea71 P 24b2931f243c419a9d1c7d00c97defac [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5de8b6fc77c4d1090acf81f46f74192" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:17:55.456707 2644 tablet_replica.cc:439] T 228e0bfc4755408f936b4539fb25ea71 P c5de8b6fc77c4d1090acf81f46f74192: TxnStatusTablet state changed. Reason: New leader 24b2931f243c419a9d1c7d00c97defac. Latest consensus state: current_term: 1 leader_uuid: "24b2931f243c419a9d1c7d00c97defac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5de8b6fc77c4d1090acf81f46f74192" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35281 } } peers { permanent_uuid: "24b2931f243c419a9d1c7d00c97defac" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46229 } } peers { permanent_uuid: "58a33666811f42fca694ee4b30acb12f" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33011 } } }
I20260618 22:17:55.456815 2644 tablet_replica.cc:442] T 228e0bfc4755408f936b4539fb25ea71 P c5de8b6fc77c4d1090acf81f46f74192: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 22:17:55.456941 2645 tablet_replica.cc:439] T 228e0bfc4755408f936b4539fb25ea71 P 58a33666811f42fca694ee4b30acb12f: TxnStatusTablet state changed. Reason: New leader 24b2931f243c419a9d1c7d00c97defac. Latest consensus state: current_term: 1 leader_uuid: "24b2931f243c419a9d1c7d00c97defac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5de8b6fc77c4d1090acf81f46f74192" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35281 } } peers { permanent_uuid: "24b2931f243c419a9d1c7d00c97defac" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46229 } } peers { permanent_uuid: "58a33666811f42fca694ee4b30acb12f" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33011 } } }
I20260618 22:17:55.457043 2645 tablet_replica.cc:442] T 228e0bfc4755408f936b4539fb25ea71 P 58a33666811f42fca694ee4b30acb12f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 22:17:55.459717 2643 tablet_replica.cc:439] T 228e0bfc4755408f936b4539fb25ea71 P 24b2931f243c419a9d1c7d00c97defac: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "24b2931f243c419a9d1c7d00c97defac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5de8b6fc77c4d1090acf81f46f74192" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35281 } } peers { permanent_uuid: "24b2931f243c419a9d1c7d00c97defac" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46229 } } peers { permanent_uuid: "58a33666811f42fca694ee4b30acb12f" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33011 } } }
I20260618 22:17:55.459826 2643 tablet_replica.cc:442] T 228e0bfc4755408f936b4539fb25ea71 P 24b2931f243c419a9d1c7d00c97defac: This TxnStatusTablet replica's current role is: LEADER
I20260618 22:17:55.459942 2643 tablet_replica.cc:439] T 228e0bfc4755408f936b4539fb25ea71 P 24b2931f243c419a9d1c7d00c97defac: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "24b2931f243c419a9d1c7d00c97defac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5de8b6fc77c4d1090acf81f46f74192" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35281 } } peers { permanent_uuid: "24b2931f243c419a9d1c7d00c97defac" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46229 } } peers { permanent_uuid: "58a33666811f42fca694ee4b30acb12f" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33011 } } }
I20260618 22:17:55.460012 2643 tablet_replica.cc:442] T 228e0bfc4755408f936b4539fb25ea71 P 24b2931f243c419a9d1c7d00c97defac: This TxnStatusTablet replica's current role is: LEADER
I20260618 22:17:55.460304 2644 tablet_replica.cc:439] T 228e0bfc4755408f936b4539fb25ea71 P c5de8b6fc77c4d1090acf81f46f74192: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "24b2931f243c419a9d1c7d00c97defac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5de8b6fc77c4d1090acf81f46f74192" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35281 } } peers { permanent_uuid: "24b2931f243c419a9d1c7d00c97defac" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46229 } } peers { permanent_uuid: "58a33666811f42fca694ee4b30acb12f" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33011 } } }
I20260618 22:17:55.460421 2644 tablet_replica.cc:442] T 228e0bfc4755408f936b4539fb25ea71 P c5de8b6fc77c4d1090acf81f46f74192: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 22:17:55.460728 2663 mvcc.cc:205] Tried to move back new op lower bound from 7298339125044301824 to 7298339124958998528. Current Snapshot: MvccSnapshot[applied={T|T < 7298339125044301824}]
I20260618 22:17:55.461810 2645 tablet_replica.cc:439] T 228e0bfc4755408f936b4539fb25ea71 P 58a33666811f42fca694ee4b30acb12f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "24b2931f243c419a9d1c7d00c97defac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5de8b6fc77c4d1090acf81f46f74192" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35281 } } peers { permanent_uuid: "24b2931f243c419a9d1c7d00c97defac" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46229 } } peers { permanent_uuid: "58a33666811f42fca694ee4b30acb12f" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33011 } } }
I20260618 22:17:55.461930 2645 tablet_replica.cc:442] T 228e0bfc4755408f936b4539fb25ea71 P 58a33666811f42fca694ee4b30acb12f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 22:17:55.473809 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:55.470921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40926 (local address 127.0.98.66:46229)
0618 22:17:55.471042 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0618 22:17:55.471048 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:17:55.471062 (+ 14us) server_negotiation.cc:407] Connection header received
0618 22:17:55.471113 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:17:55.471115 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:17:55.471150 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:17:55.471226 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:17:55.471479 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:55.471967 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:55.472833 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:55.473045 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:55.473150 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:55.473403 (+ 253us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:17:55.473468 (+ 65us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:17:55.473545 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0618 22:17:55.473632 (+ 87us) server_negotiation.cc:299] Negotiation successful
0618 22:17:55.473673 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260618 22:17:55.483955 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:55.480513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:52793 (local address 127.0.98.66:46229)
0618 22:17:55.480633 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0618 22:17:55.480637 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:17:55.480842 (+ 205us) server_negotiation.cc:407] Connection header received
0618 22:17:55.480944 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:17:55.480953 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:17:55.480991 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:17:55.481049 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:17:55.481294 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:55.481739 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:55.482986 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:55.483180 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:55.483258 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:55.483570 (+ 312us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:17:55.483626 (+ 56us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:17:55.483677 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0618 22:17:55.483798 (+ 121us) server_negotiation.cc:299] Negotiation successful
0618 22:17:55.483838 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260618 22:17:55.484146 2674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:55.480462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:46229 (local address 127.0.98.66:52793)
0618 22:17:55.480757 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0618 22:17:55.480772 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:17:55.480867 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:17:55.481080 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:17:55.481083 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:17:55.481090 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0618 22:17:55.481198 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:55.481203 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:55.481851 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:55.481855 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:55.482882 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:55.482891 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:55.483290 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:55.483294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:55.483386 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:55.483431 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0618 22:17:55.483708 (+ 277us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0618 22:17:55.483714 (+ 6us) client_negotiation.cc:769] Sending connection context
0618 22:17:55.483781 (+ 67us) client_negotiation.cc:240] Negotiation successful
0618 22:17:55.483864 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202,"mutex_wait_us":3,"thread_start_us":113,"threads_started":1}
I20260618 22:17:55.490782 2288 raft_consensus.cc:1261] T ae86fe5b4486438680bf8aef5de578dc P c5de8b6fc77c4d1090acf81f46f74192 [term 1 FOLLOWER]: Refusing update from remote peer 24b2931f243c419a9d1c7d00c97defac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:17:55.490864 2568 raft_consensus.cc:1261] T ae86fe5b4486438680bf8aef5de578dc P 58a33666811f42fca694ee4b30acb12f [term 1 FOLLOWER]: Refusing update from remote peer 24b2931f243c419a9d1c7d00c97defac: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:17:55.491241 2667 consensus_queue.cc:1035] T ae86fe5b4486438680bf8aef5de578dc P 24b2931f243c419a9d1c7d00c97defac [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5de8b6fc77c4d1090acf81f46f74192" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:17:55.491401 2647 consensus_queue.cc:1035] T ae86fe5b4486438680bf8aef5de578dc P 24b2931f243c419a9d1c7d00c97defac [LEADER]: Connected to new peer: Peer: permanent_uuid: "58a33666811f42fca694ee4b30acb12f" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:17:55.494195 2662 mvcc.cc:205] Tried to move back new op lower bound from 7298339125207117824 to 7298339123625205760. Current Snapshot: MvccSnapshot[applied={T|T < 7298339125207117824}]
I20260618 22:17:55.531723 2130 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:17:55.531914 2130 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:17:55.534215 2130 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c640d89b20c44a6cb8af3a75a11e252a: Sending DeleteTablet for 3 replicas of tablet ae86fe5b4486438680bf8aef5de578dc
I20260618 22:17:55.534945 2268 tablet_service.cc:1514] Processing DeleteTablet for tablet ae86fe5b4486438680bf8aef5de578dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:17:55 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:39364
I20260618 22:17:55.535041 2548 tablet_service.cc:1514] Processing DeleteTablet for tablet ae86fe5b4486438680bf8aef5de578dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:17:55 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:48362
I20260618 22:17:55.535431 2678 tablet_replica.cc:330] T ae86fe5b4486438680bf8aef5de578dc P c5de8b6fc77c4d1090acf81f46f74192: stopping tablet replica
I20260618 22:17:55.535408 2401 tablet_service.cc:1514] Processing DeleteTablet for tablet ae86fe5b4486438680bf8aef5de578dc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:17:55 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:40906
I20260618 22:17:55.535786 2678 raft_consensus.cc:2229] T ae86fe5b4486438680bf8aef5de578dc P c5de8b6fc77c4d1090acf81f46f74192 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:17:55.535808 2679 tablet_replica.cc:330] T ae86fe5b4486438680bf8aef5de578dc P 58a33666811f42fca694ee4b30acb12f: stopping tablet replica
I20260618 22:17:55.535885 2130 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38894:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260618 22:17:55.535984 2678 raft_consensus.cc:2258] T ae86fe5b4486438680bf8aef5de578dc P c5de8b6fc77c4d1090acf81f46f74192 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:17:55.536090 2679 raft_consensus.cc:2229] T ae86fe5b4486438680bf8aef5de578dc P 58a33666811f42fca694ee4b30acb12f [term 1 FOLLOWER]: Raft consensus shutting down.
W20260618 22:17:55.536238 2130 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:17:55.536260 2679 raft_consensus.cc:2258] T ae86fe5b4486438680bf8aef5de578dc P 58a33666811f42fca694ee4b30acb12f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:17:55.537299 2678 ts_tablet_manager.cc:1905] T ae86fe5b4486438680bf8aef5de578dc P c5de8b6fc77c4d1090acf81f46f74192: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:17:55.537416 2679 ts_tablet_manager.cc:1905] T ae86fe5b4486438680bf8aef5de578dc P 58a33666811f42fca694ee4b30acb12f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:17:55.539745 2680 tablet_replica.cc:330] T ae86fe5b4486438680bf8aef5de578dc P 24b2931f243c419a9d1c7d00c97defac: stopping tablet replica
I20260618 22:17:55.540059 2680 raft_consensus.cc:2229] T ae86fe5b4486438680bf8aef5de578dc P 24b2931f243c419a9d1c7d00c97defac [term 1 LEADER]: Raft consensus shutting down.
I20260618 22:17:55.540319 2679 ts_tablet_manager.cc:1918] T ae86fe5b4486438680bf8aef5de578dc P 58a33666811f42fca694ee4b30acb12f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260618 22:17:55.540417 2679 log.cc:1196] T ae86fe5b4486438680bf8aef5de578dc P 58a33666811f42fca694ee4b30acb12f: Deleting WAL directory at /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/ts-2/wal/wals/ae86fe5b4486438680bf8aef5de578dc
I20260618 22:17:55.540417 2678 ts_tablet_manager.cc:1918] T ae86fe5b4486438680bf8aef5de578dc P c5de8b6fc77c4d1090acf81f46f74192: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260618 22:17:55.540500 2678 log.cc:1196] T ae86fe5b4486438680bf8aef5de578dc P c5de8b6fc77c4d1090acf81f46f74192: Deleting WAL directory at /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/ts-0/wal/wals/ae86fe5b4486438680bf8aef5de578dc
I20260618 22:17:55.540696 2680 raft_consensus.cc:2258] T ae86fe5b4486438680bf8aef5de578dc P 24b2931f243c419a9d1c7d00c97defac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:17:55.540751 2679 ts_tablet_manager.cc:1939] T ae86fe5b4486438680bf8aef5de578dc P 58a33666811f42fca694ee4b30acb12f: Deleting consensus metadata
I20260618 22:17:55.540915 2678 ts_tablet_manager.cc:1939] T ae86fe5b4486438680bf8aef5de578dc P c5de8b6fc77c4d1090acf81f46f74192: Deleting consensus metadata
I20260618 22:17:55.541456 2268 tablet_service.cc:1467] Processing CreateTablet for tablet 5bfd5579b8da403b804fc0bc447ddace (DEFAULT_TABLE table=test-table [id=588da2372ca044b295503967d3a4ad7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:17:55.541688 2268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bfd5579b8da403b804fc0bc447ddace. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:17:55.541646 2548 tablet_service.cc:1467] Processing CreateTablet for tablet 5bfd5579b8da403b804fc0bc447ddace (DEFAULT_TABLE table=test-table [id=588da2372ca044b295503967d3a4ad7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:17:55.541872 2548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bfd5579b8da403b804fc0bc447ddace. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:17:55.542037 2116 catalog_manager.cc:4841] TS c5de8b6fc77c4d1090acf81f46f74192 (127.0.98.65:35281): tablet ae86fe5b4486438680bf8aef5de578dc (table test-table [id=e3c661eac03c41b197429437e49b4630]) successfully deleted
I20260618 22:17:55.543447 2639 tablet_bootstrap.cc:492] T 5bfd5579b8da403b804fc0bc447ddace P c5de8b6fc77c4d1090acf81f46f74192: Bootstrap starting.
I20260618 22:17:55.543644 2638 tablet_bootstrap.cc:492] T 5bfd5579b8da403b804fc0bc447ddace P 58a33666811f42fca694ee4b30acb12f: Bootstrap starting.
I20260618 22:17:55.544299 2639 tablet_bootstrap.cc:654] T 5bfd5579b8da403b804fc0bc447ddace P c5de8b6fc77c4d1090acf81f46f74192: Neither blocks nor log segments found. Creating new log.
I20260618 22:17:55.544286 2118 catalog_manager.cc:4841] TS 58a33666811f42fca694ee4b30acb12f (127.0.98.67:33011): tablet ae86fe5b4486438680bf8aef5de578dc (table test-table [id=e3c661eac03c41b197429437e49b4630]) successfully deleted
I20260618 22:17:55.544528 2638 tablet_bootstrap.cc:654] T 5bfd5579b8da403b804fc0bc447ddace P 58a33666811f42fca694ee4b30acb12f: Neither blocks nor log segments found. Creating new log.
I20260618 22:17:55.544963 2401 tablet_service.cc:1467] Processing CreateTablet for tablet 5bfd5579b8da403b804fc0bc447ddace (DEFAULT_TABLE table=test-table [id=588da2372ca044b295503967d3a4ad7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:17:55.545199 2401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5bfd5579b8da403b804fc0bc447ddace. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:17:55.545774 2638 tablet_bootstrap.cc:492] T 5bfd5579b8da403b804fc0bc447ddace P 58a33666811f42fca694ee4b30acb12f: No bootstrap required, opened a new log
I20260618 22:17:55.545852 2638 ts_tablet_manager.cc:1397] T 5bfd5579b8da403b804fc0bc447ddace P 58a33666811f42fca694ee4b30acb12f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260618 22:17:55.546281 2638 raft_consensus.cc:348] T 5bfd5579b8da403b804fc0bc447ddace P 58a33666811f42fca694ee4b30acb12f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5de8b6fc77c4d1090acf81f46f74192" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35281 } } peers { permanent_uuid: "24b2931f243c419a9d1c7d00c97defac" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46229 } } peers { permanent_uuid: "58a33666811f42fca694ee4b30acb12f" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33011 } }
I20260618 22:17:55.546412 2638 raft_consensus.cc:374] T 5bfd5579b8da403b804fc0bc447ddace P 58a33666811f42fca694ee4b30acb12f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:17:55.546445 2638 raft_consensus.cc:729] T 5bfd5579b8da403b804fc0bc447ddace P 58a33666811f42fca694ee4b30acb12f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58a33666811f42fca694ee4b30acb12f, State: Initialized, Role: FOLLOWER
I20260618 22:17:55.546450 2680 ts_tablet_manager.cc:1905] T ae86fe5b4486438680bf8aef5de578dc P 24b2931f243c419a9d1c7d00c97defac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:17:55.546562 2638 consensus_queue.cc:260] T 5bfd5579b8da403b804fc0bc447ddace P 58a33666811f42fca694ee4b30acb12f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5de8b6fc77c4d1090acf81f46f74192" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35281 } } peers { permanent_uuid: "24b2931f243c419a9d1c7d00c97defac" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46229 } } peers { permanent_uuid: "58a33666811f42fca694ee4b30acb12f" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33011 } }
I20260618 22:17:55.546787 2638 ts_tablet_manager.cc:1428] T 5bfd5579b8da403b804fc0bc447ddace P 58a33666811f42fca694ee4b30acb12f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260618 22:17:55.547396 2637 tablet_bootstrap.cc:492] T 5bfd5579b8da403b804fc0bc447ddace P 24b2931f243c419a9d1c7d00c97defac: Bootstrap starting.
I20260618 22:17:55.548167 2639 tablet_bootstrap.cc:492] T 5bfd5579b8da403b804fc0bc447ddace P c5de8b6fc77c4d1090acf81f46f74192: No bootstrap required, opened a new log
I20260618 22:17:55.548239 2639 ts_tablet_manager.cc:1397] T 5bfd5579b8da403b804fc0bc447ddace P c5de8b6fc77c4d1090acf81f46f74192: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260618 22:17:55.548334 2637 tablet_bootstrap.cc:654] T 5bfd5579b8da403b804fc0bc447ddace P 24b2931f243c419a9d1c7d00c97defac: Neither blocks nor log segments found. Creating new log.
I20260618 22:17:55.548686 2639 raft_consensus.cc:348] T 5bfd5579b8da403b804fc0bc447ddace P c5de8b6fc77c4d1090acf81f46f74192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5de8b6fc77c4d1090acf81f46f74192" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35281 } } peers { permanent_uuid: "24b2931f243c419a9d1c7d00c97defac" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46229 } } peers { permanent_uuid: "58a33666811f42fca694ee4b30acb12f" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33011 } }
I20260618 22:17:55.548789 2639 raft_consensus.cc:374] T 5bfd5579b8da403b804fc0bc447ddace P c5de8b6fc77c4d1090acf81f46f74192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:17:55.548820 2639 raft_consensus.cc:729] T 5bfd5579b8da403b804fc0bc447ddace P c5de8b6fc77c4d1090acf81f46f74192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5de8b6fc77c4d1090acf81f46f74192, State: Initialized, Role: FOLLOWER
I20260618 22:17:55.548923 2639 consensus_queue.cc:260] T 5bfd5579b8da403b804fc0bc447ddace P c5de8b6fc77c4d1090acf81f46f74192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5de8b6fc77c4d1090acf81f46f74192" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35281 } } peers { permanent_uuid: "24b2931f243c419a9d1c7d00c97defac" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46229 } } peers { permanent_uuid: "58a33666811f42fca694ee4b30acb12f" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33011 } }
I20260618 22:17:55.549147 2639 ts_tablet_manager.cc:1428] T 5bfd5579b8da403b804fc0bc447ddace P c5de8b6fc77c4d1090acf81f46f74192: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260618 22:17:55.549225 2680 ts_tablet_manager.cc:1918] T ae86fe5b4486438680bf8aef5de578dc P 24b2931f243c419a9d1c7d00c97defac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260618 22:17:55.549317 2680 log.cc:1196] T ae86fe5b4486438680bf8aef5de578dc P 24b2931f243c419a9d1c7d00c97defac: Deleting WAL directory at /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/ts-1/wal/wals/ae86fe5b4486438680bf8aef5de578dc
I20260618 22:17:55.549793 2680 ts_tablet_manager.cc:1939] T ae86fe5b4486438680bf8aef5de578dc P 24b2931f243c419a9d1c7d00c97defac: Deleting consensus metadata
I20260618 22:17:55.550487 2637 tablet_bootstrap.cc:492] T 5bfd5579b8da403b804fc0bc447ddace P 24b2931f243c419a9d1c7d00c97defac: No bootstrap required, opened a new log
I20260618 22:17:55.550570 2637 ts_tablet_manager.cc:1397] T 5bfd5579b8da403b804fc0bc447ddace P 24b2931f243c419a9d1c7d00c97defac: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260618 22:17:55.550906 2637 raft_consensus.cc:348] T 5bfd5579b8da403b804fc0bc447ddace P 24b2931f243c419a9d1c7d00c97defac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5de8b6fc77c4d1090acf81f46f74192" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35281 } } peers { permanent_uuid: "24b2931f243c419a9d1c7d00c97defac" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46229 } } peers { permanent_uuid: "58a33666811f42fca694ee4b30acb12f" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33011 } }
I20260618 22:17:55.550994 2637 raft_consensus.cc:374] T 5bfd5579b8da403b804fc0bc447ddace P 24b2931f243c419a9d1c7d00c97defac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:17:55.551015 2637 raft_consensus.cc:729] T 5bfd5579b8da403b804fc0bc447ddace P 24b2931f243c419a9d1c7d00c97defac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24b2931f243c419a9d1c7d00c97defac, State: Initialized, Role: FOLLOWER
I20260618 22:17:55.551155 2637 consensus_queue.cc:260] T 5bfd5579b8da403b804fc0bc447ddace P 24b2931f243c419a9d1c7d00c97defac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5de8b6fc77c4d1090acf81f46f74192" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35281 } } peers { permanent_uuid: "24b2931f243c419a9d1c7d00c97defac" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46229 } } peers { permanent_uuid: "58a33666811f42fca694ee4b30acb12f" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33011 } }
I20260618 22:17:55.551440 2637 ts_tablet_manager.cc:1428] T 5bfd5579b8da403b804fc0bc447ddace P 24b2931f243c419a9d1c7d00c97defac: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260618 22:17:55.551705 2115 catalog_manager.cc:4841] TS 24b2931f243c419a9d1c7d00c97defac (127.0.98.66:46229): tablet ae86fe5b4486438680bf8aef5de578dc (table test-table [id=e3c661eac03c41b197429437e49b4630]) successfully deleted
I20260618 22:17:55.603672 2644 raft_consensus.cc:482] T 5bfd5579b8da403b804fc0bc447ddace P c5de8b6fc77c4d1090acf81f46f74192 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:17:55.603808 2644 raft_consensus.cc:504] T 5bfd5579b8da403b804fc0bc447ddace P c5de8b6fc77c4d1090acf81f46f74192 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5de8b6fc77c4d1090acf81f46f74192" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35281 } } peers { permanent_uuid: "24b2931f243c419a9d1c7d00c97defac" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46229 } } peers { permanent_uuid: "58a33666811f42fca694ee4b30acb12f" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33011 } }
I20260618 22:17:55.604928 2644 leader_election.cc:288] T 5bfd5579b8da403b804fc0bc447ddace P c5de8b6fc77c4d1090acf81f46f74192 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 24b2931f243c419a9d1c7d00c97defac (127.0.98.66:46229), 58a33666811f42fca694ee4b30acb12f (127.0.98.67:33011)
I20260618 22:17:55.609156 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:55.605206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:48365 (local address 127.0.98.66:46229)
0618 22:17:55.605354 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0618 22:17:55.605359 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:17:55.605481 (+ 122us) server_negotiation.cc:407] Connection header received
0618 22:17:55.605694 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:17:55.605697 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:17:55.605758 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:17:55.605808 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:17:55.606299 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:55.606795 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:55.608116 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:55.608621 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:55.608725 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:55.608788 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0618 22:17:55.608969 (+ 181us) server_negotiation.cc:299] Negotiation successful
0618 22:17:55.609035 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260618 22:17:55.609184 2684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:55.605115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:46229 (local address 127.0.98.65:48365)
0618 22:17:55.605399 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0618 22:17:55.605414 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:17:55.605563 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:17:55.606002 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:17:55.606008 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:17:55.606030 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:17:55.606180 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:55.606186 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:55.606910 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:55.606916 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:55.607994 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:55.608003 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:55.608735 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:55.608739 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:55.608839 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:55.608866 (+ 27us) client_negotiation.cc:769] Sending connection context
0618 22:17:55.608944 (+ 78us) client_negotiation.cc:240] Negotiation successful
0618 22:17:55.609015 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":30,"thread_start_us":63,"threads_started":1}
I20260618 22:17:55.609720 2685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:55.605173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:33011 (local address 127.0.98.65:56689)
0618 22:17:55.605497 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0618 22:17:55.605510 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:17:55.605583 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:17:55.606047 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:17:55.606050 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:17:55.606073 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:17:55.606186 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:55.606192 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:55.607322 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:55.607325 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:55.608406 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:55.608416 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:55.609289 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:55.609293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:55.609408 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:55.609426 (+ 18us) client_negotiation.cc:769] Sending connection context
0618 22:17:55.609503 (+ 77us) client_negotiation.cc:240] Negotiation successful
0618 22:17:55.609578 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":3,"thread_start_us":59,"threads_started":1}
I20260618 22:17:55.610044 2428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bfd5579b8da403b804fc0bc447ddace" candidate_uuid: "c5de8b6fc77c4d1090acf81f46f74192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24b2931f243c419a9d1c7d00c97defac" is_pre_election: true
I20260618 22:17:55.610291 2428 raft_consensus.cc:2454] T 5bfd5579b8da403b804fc0bc447ddace P 24b2931f243c419a9d1c7d00c97defac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5de8b6fc77c4d1090acf81f46f74192 in term 0.
I20260618 22:17:55.610354 2686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:55.605494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:56689 (local address 127.0.98.67:33011)
0618 22:17:55.605779 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0618 22:17:55.605785 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:17:55.605801 (+ 16us) server_negotiation.cc:407] Connection header received
0618 22:17:55.605848 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:17:55.605851 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:17:55.605942 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:17:55.606024 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:17:55.606317 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:55.607198 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:55.608536 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:55.609145 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:55.610049 (+ 904us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:55.610116 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0618 22:17:55.610155 (+ 39us) server_negotiation.cc:299] Negotiation successful
0618 22:17:55.610227 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":64,"threads_started":1}
I20260618 22:17:55.610672 2568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bfd5579b8da403b804fc0bc447ddace" candidate_uuid: "c5de8b6fc77c4d1090acf81f46f74192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58a33666811f42fca694ee4b30acb12f" is_pre_election: true
I20260618 22:17:55.610703 2219 leader_election.cc:302] T 5bfd5579b8da403b804fc0bc447ddace P c5de8b6fc77c4d1090acf81f46f74192 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24b2931f243c419a9d1c7d00c97defac, c5de8b6fc77c4d1090acf81f46f74192; no voters:
I20260618 22:17:55.610828 2568 raft_consensus.cc:2454] T 5bfd5579b8da403b804fc0bc447ddace P 58a33666811f42fca694ee4b30acb12f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5de8b6fc77c4d1090acf81f46f74192 in term 0.
I20260618 22:17:55.610958 2644 raft_consensus.cc:2781] T 5bfd5579b8da403b804fc0bc447ddace P c5de8b6fc77c4d1090acf81f46f74192 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:17:55.611014 2644 raft_consensus.cc:482] T 5bfd5579b8da403b804fc0bc447ddace P c5de8b6fc77c4d1090acf81f46f74192 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:17:55.611053 2644 raft_consensus.cc:3037] T 5bfd5579b8da403b804fc0bc447ddace P c5de8b6fc77c4d1090acf81f46f74192 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:17:55.612329 2644 raft_consensus.cc:504] T 5bfd5579b8da403b804fc0bc447ddace P c5de8b6fc77c4d1090acf81f46f74192 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5de8b6fc77c4d1090acf81f46f74192" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35281 } } peers { permanent_uuid: "24b2931f243c419a9d1c7d00c97defac" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46229 } } peers { permanent_uuid: "58a33666811f42fca694ee4b30acb12f" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33011 } }
I20260618 22:17:55.612756 2644 leader_election.cc:288] T 5bfd5579b8da403b804fc0bc447ddace P c5de8b6fc77c4d1090acf81f46f74192 [CANDIDATE]: Term 1 election: Requested vote from peers 24b2931f243c419a9d1c7d00c97defac (127.0.98.66:46229), 58a33666811f42fca694ee4b30acb12f (127.0.98.67:33011)
I20260618 22:17:55.613158 2568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bfd5579b8da403b804fc0bc447ddace" candidate_uuid: "c5de8b6fc77c4d1090acf81f46f74192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58a33666811f42fca694ee4b30acb12f"
I20260618 22:17:55.613279 2568 raft_consensus.cc:3037] T 5bfd5579b8da403b804fc0bc447ddace P 58a33666811f42fca694ee4b30acb12f [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:17:55.614298 2568 raft_consensus.cc:2454] T 5bfd5579b8da403b804fc0bc447ddace P 58a33666811f42fca694ee4b30acb12f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5de8b6fc77c4d1090acf81f46f74192 in term 1.
I20260618 22:17:55.614621 2222 leader_election.cc:302] T 5bfd5579b8da403b804fc0bc447ddace P c5de8b6fc77c4d1090acf81f46f74192 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 58a33666811f42fca694ee4b30acb12f, c5de8b6fc77c4d1090acf81f46f74192; no voters:
I20260618 22:17:55.614670 2428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5bfd5579b8da403b804fc0bc447ddace" candidate_uuid: "c5de8b6fc77c4d1090acf81f46f74192" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24b2931f243c419a9d1c7d00c97defac"
I20260618 22:17:55.614784 2644 raft_consensus.cc:2781] T 5bfd5579b8da403b804fc0bc447ddace P c5de8b6fc77c4d1090acf81f46f74192 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:17:55.614827 2428 raft_consensus.cc:3037] T 5bfd5579b8da403b804fc0bc447ddace P 24b2931f243c419a9d1c7d00c97defac [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:17:55.614974 2644 raft_consensus.cc:686] T 5bfd5579b8da403b804fc0bc447ddace P c5de8b6fc77c4d1090acf81f46f74192 [term 1 LEADER]: Becoming Leader. State: Replica: c5de8b6fc77c4d1090acf81f46f74192, State: Running, Role: LEADER
I20260618 22:17:55.615208 2644 consensus_queue.cc:237] T 5bfd5579b8da403b804fc0bc447ddace P c5de8b6fc77c4d1090acf81f46f74192 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5de8b6fc77c4d1090acf81f46f74192" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35281 } } peers { permanent_uuid: "24b2931f243c419a9d1c7d00c97defac" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46229 } } peers { permanent_uuid: "58a33666811f42fca694ee4b30acb12f" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33011 } }
I20260618 22:17:55.616037 2428 raft_consensus.cc:2454] T 5bfd5579b8da403b804fc0bc447ddace P 24b2931f243c419a9d1c7d00c97defac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5de8b6fc77c4d1090acf81f46f74192 in term 1.
I20260618 22:17:55.617486 2131 catalog_manager.cc:5495] T 5bfd5579b8da403b804fc0bc447ddace P c5de8b6fc77c4d1090acf81f46f74192 reported cstate change: term changed from 0 to 1, leader changed from <none> to c5de8b6fc77c4d1090acf81f46f74192 (127.0.98.65). New cstate: current_term: 1 leader_uuid: "c5de8b6fc77c4d1090acf81f46f74192" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5de8b6fc77c4d1090acf81f46f74192" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35281 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "24b2931f243c419a9d1c7d00c97defac" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58a33666811f42fca694ee4b30acb12f" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33011 } health_report { overall_health: UNKNOWN } } }
I20260618 22:17:55.635250 2689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:55.631175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39370 (local address 127.0.98.65:35281)
0618 22:17:55.631418 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0618 22:17:55.631425 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:17:55.631451 (+ 26us) server_negotiation.cc:407] Connection header received
0618 22:17:55.631506 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:17:55.631508 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:17:55.631554 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:17:55.631643 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:17:55.631883 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:55.632386 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:55.633513 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:55.633730 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:55.633800 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:55.634086 (+ 286us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:17:55.634559 (+ 473us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:17:55.634648 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0618 22:17:55.634765 (+ 117us) server_negotiation.cc:299] Negotiation successful
0618 22:17:55.634840 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":116,"threads_started":1}
I20260618 22:17:55.646431 2692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:55.641951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:46229 (local address 127.0.98.65:36151)
0618 22:17:55.642785 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0618 22:17:55.642800 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:17:55.642912 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:17:55.643262 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:17:55.643266 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:17:55.643278 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0618 22:17:55.643428 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:55.643442 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:55.644506 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:55.644509 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:55.645163 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:55.645171 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:55.645734 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:55.645736 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:55.645805 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:55.645841 (+ 36us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0618 22:17:55.646191 (+ 350us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0618 22:17:55.646196 (+ 5us) client_negotiation.cc:769] Sending connection context
0618 22:17:55.646246 (+ 50us) client_negotiation.cc:240] Negotiation successful
0618 22:17:55.646309 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":93,"threads_started":1}
I20260618 22:17:55.646441 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:55.642100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:36151 (local address 127.0.98.66:46229)
0618 22:17:55.642291 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0618 22:17:55.642296 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:17:55.642896 (+ 600us) server_negotiation.cc:407] Connection header received
0618 22:17:55.643026 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:17:55.643029 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:17:55.643092 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:17:55.643196 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:17:55.643573 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:55.644400 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:55.645274 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:55.645615 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:55.645716 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:55.646029 (+ 313us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:17:55.646106 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:17:55.646179 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0618 22:17:55.646273 (+ 94us) server_negotiation.cc:299] Negotiation successful
0618 22:17:55.646319 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260618 22:17:55.652047 2568 raft_consensus.cc:1261] T 5bfd5579b8da403b804fc0bc447ddace P 58a33666811f42fca694ee4b30acb12f [term 1 FOLLOWER]: Refusing update from remote peer c5de8b6fc77c4d1090acf81f46f74192: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:17:55.652253 2428 raft_consensus.cc:1261] T 5bfd5579b8da403b804fc0bc447ddace P 24b2931f243c419a9d1c7d00c97defac [term 1 FOLLOWER]: Refusing update from remote peer c5de8b6fc77c4d1090acf81f46f74192: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:17:55.652529 2687 consensus_queue.cc:1035] T 5bfd5579b8da403b804fc0bc447ddace P c5de8b6fc77c4d1090acf81f46f74192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "58a33666811f42fca694ee4b30acb12f" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:17:55.652705 2644 consensus_queue.cc:1035] T 5bfd5579b8da403b804fc0bc447ddace P c5de8b6fc77c4d1090acf81f46f74192 [LEADER]: Connected to new peer: Peer: permanent_uuid: "24b2931f243c419a9d1c7d00c97defac" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:17:55.671120 2674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:55.667839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:35281 (local address 127.0.98.66:39781)
0618 22:17:55.667986 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0618 22:17:55.668001 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:17:55.668070 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:17:55.668340 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:17:55.668343 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:17:55.668351 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0618 22:17:55.668446 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:55.668452 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:55.669330 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:55.669336 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:55.669983 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:55.669999 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:55.670447 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:55.670450 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:55.670538 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:55.670570 (+ 32us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0618 22:17:55.670848 (+ 278us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0618 22:17:55.670857 (+ 9us) client_negotiation.cc:769] Sending connection context
0618 22:17:55.670931 (+ 74us) client_negotiation.cc:240] Negotiation successful
0618 22:17:55.671005 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260618 22:17:55.671120 2689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:55.667895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:39781 (local address 127.0.98.65:35281)
0618 22:17:55.668049 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0618 22:17:55.668053 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:17:55.668068 (+ 15us) server_negotiation.cc:407] Connection header received
0618 22:17:55.668181 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:17:55.668184 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:17:55.668244 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:17:55.668319 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:17:55.668562 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:55.669226 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:55.670100 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:55.670343 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:55.670452 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:55.670675 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:17:55.670756 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:17:55.670828 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0618 22:17:55.670944 (+ 116us) server_negotiation.cc:299] Negotiation successful
0618 22:17:55.670995 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
W20260618 22:17:55.672025 2663 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260618 22:17:57.233100 2130 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:17:57.233275 2130 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38894:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:17:57.235594 2130 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c640d89b20c44a6cb8af3a75a11e252a: Sending DeleteTablet for 3 replicas of tablet 5bfd5579b8da403b804fc0bc447ddace
I20260618 22:17:57.236099 2268 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bfd5579b8da403b804fc0bc447ddace with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:17:57 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:39364
I20260618 22:17:57.236265 2401 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bfd5579b8da403b804fc0bc447ddace with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:17:57 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:40906
I20260618 22:17:57.236265 2548 tablet_service.cc:1514] Processing DeleteTablet for tablet 5bfd5579b8da403b804fc0bc447ddace with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:17:57 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:48362
I20260618 22:17:57.236431 2720 tablet_replica.cc:330] T 5bfd5579b8da403b804fc0bc447ddace P c5de8b6fc77c4d1090acf81f46f74192: stopping tablet replica
I20260618 22:17:57.236542 2720 raft_consensus.cc:2229] T 5bfd5579b8da403b804fc0bc447ddace P c5de8b6fc77c4d1090acf81f46f74192 [term 1 LEADER]: Raft consensus shutting down.
I20260618 22:17:57.236608 2722 tablet_replica.cc:330] T 5bfd5579b8da403b804fc0bc447ddace P 58a33666811f42fca694ee4b30acb12f: stopping tablet replica
I20260618 22:17:57.236734 2722 raft_consensus.cc:2229] T 5bfd5579b8da403b804fc0bc447ddace P 58a33666811f42fca694ee4b30acb12f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:17:57.236869 2722 raft_consensus.cc:2258] T 5bfd5579b8da403b804fc0bc447ddace P 58a33666811f42fca694ee4b30acb12f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:17:57.237130 2721 tablet_replica.cc:330] T 5bfd5579b8da403b804fc0bc447ddace P 24b2931f243c419a9d1c7d00c97defac: stopping tablet replica
I20260618 22:17:57.237250 2721 raft_consensus.cc:2229] T 5bfd5579b8da403b804fc0bc447ddace P 24b2931f243c419a9d1c7d00c97defac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:17:57.237375 2721 raft_consensus.cc:2258] T 5bfd5579b8da403b804fc0bc447ddace P 24b2931f243c419a9d1c7d00c97defac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:17:57.237819 2722 ts_tablet_manager.cc:1905] T 5bfd5579b8da403b804fc0bc447ddace P 58a33666811f42fca694ee4b30acb12f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:17:57.239180 2720 raft_consensus.cc:2258] T 5bfd5579b8da403b804fc0bc447ddace P c5de8b6fc77c4d1090acf81f46f74192 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:17:57.240005 2720 ts_tablet_manager.cc:1905] T 5bfd5579b8da403b804fc0bc447ddace P c5de8b6fc77c4d1090acf81f46f74192: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:17:57.241219 2721 ts_tablet_manager.cc:1905] T 5bfd5579b8da403b804fc0bc447ddace P 24b2931f243c419a9d1c7d00c97defac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:17:57.242404 2722 ts_tablet_manager.cc:1918] T 5bfd5579b8da403b804fc0bc447ddace P 58a33666811f42fca694ee4b30acb12f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260618 22:17:57.242512 2722 log.cc:1196] T 5bfd5579b8da403b804fc0bc447ddace P 58a33666811f42fca694ee4b30acb12f: Deleting WAL directory at /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/ts-2/wal/wals/5bfd5579b8da403b804fc0bc447ddace
I20260618 22:17:57.242827 2722 ts_tablet_manager.cc:1939] T 5bfd5579b8da403b804fc0bc447ddace P 58a33666811f42fca694ee4b30acb12f: Deleting consensus metadata
I20260618 22:17:57.243006 2720 ts_tablet_manager.cc:1918] T 5bfd5579b8da403b804fc0bc447ddace P c5de8b6fc77c4d1090acf81f46f74192: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260618 22:17:57.243171 2720 log.cc:1196] T 5bfd5579b8da403b804fc0bc447ddace P c5de8b6fc77c4d1090acf81f46f74192: Deleting WAL directory at /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/ts-0/wal/wals/5bfd5579b8da403b804fc0bc447ddace
I20260618 22:17:57.243475 2720 ts_tablet_manager.cc:1939] T 5bfd5579b8da403b804fc0bc447ddace P c5de8b6fc77c4d1090acf81f46f74192: Deleting consensus metadata
I20260618 22:17:57.243534 2118 catalog_manager.cc:4841] TS 58a33666811f42fca694ee4b30acb12f (127.0.98.67:33011): tablet 5bfd5579b8da403b804fc0bc447ddace (table test-table [id=588da2372ca044b295503967d3a4ad7e]) successfully deleted
I20260618 22:17:57.244122 2721 ts_tablet_manager.cc:1918] T 5bfd5579b8da403b804fc0bc447ddace P 24b2931f243c419a9d1c7d00c97defac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260618 22:17:57.244122 2116 catalog_manager.cc:4841] TS c5de8b6fc77c4d1090acf81f46f74192 (127.0.98.65:35281): tablet 5bfd5579b8da403b804fc0bc447ddace (table test-table [id=588da2372ca044b295503967d3a4ad7e]) successfully deleted
I20260618 22:17:57.244202 2721 log.cc:1196] T 5bfd5579b8da403b804fc0bc447ddace P 24b2931f243c419a9d1c7d00c97defac: Deleting WAL directory at /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/ts-1/wal/wals/5bfd5579b8da403b804fc0bc447ddace
I20260618 22:17:57.244462 2721 ts_tablet_manager.cc:1939] T 5bfd5579b8da403b804fc0bc447ddace P 24b2931f243c419a9d1c7d00c97defac: Deleting consensus metadata
I20260618 22:17:57.245043 2115 catalog_manager.cc:4841] TS 24b2931f243c419a9d1c7d00c97defac (127.0.98.66:46229): tablet 5bfd5579b8da403b804fc0bc447ddace (table test-table [id=588da2372ca044b295503967d3a4ad7e]) successfully deleted
Jun 18 22:17:57 dist-test-slave-g6r4 krb5kdc[2086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821077, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 18 22:17:57 dist-test-slave-g6r4 krb5kdc[2086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821077, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:17:57.268420 2730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:57.255092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38912 (local address 127.0.98.126:46833)
0618 22:17:57.255323 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0618 22:17:57.255329 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:17:57.255342 (+ 13us) server_negotiation.cc:407] Connection header received
0618 22:17:57.255445 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:17:57.255448 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:17:57.255488 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:17:57.255564 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:17:57.256276 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:57.256868 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:57.257697 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:57.257998 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:57.258086 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:57.265098 (+ 7012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:17:57.265101 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:17:57.265103 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:17:57.265137 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:17:57.266737 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:17:57.267201 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:17:57.267204 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:17:57.267205 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:17:57.267284 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:17:57.267570 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:17:57.267577 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:17:57.267578 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:17:57.267746 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:17:57.267848 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0618 22:17:57.268082 (+ 234us) server_negotiation.cc:299] Negotiation successful
0618 22:17:57.268229 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":98,"threads_started":1}
I20260618 22:17:57.270913 2130 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38912:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 22:17:57.271266 2130 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:17:57.275735 2268 tablet_service.cc:1467] Processing CreateTablet for tablet d0114939b42040328f18490e90dd67ef (DEFAULT_TABLE table=test-table [id=359f690f3feb44d28ce647683e0c2cb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:17:57.275816 2401 tablet_service.cc:1467] Processing CreateTablet for tablet d0114939b42040328f18490e90dd67ef (DEFAULT_TABLE table=test-table [id=359f690f3feb44d28ce647683e0c2cb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:17:57.276054 2268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0114939b42040328f18490e90dd67ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:17:57.276082 2401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0114939b42040328f18490e90dd67ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:17:57.277894 2731 tablet_bootstrap.cc:492] T d0114939b42040328f18490e90dd67ef P c5de8b6fc77c4d1090acf81f46f74192: Bootstrap starting.
I20260618 22:17:57.277930 2732 tablet_bootstrap.cc:492] T d0114939b42040328f18490e90dd67ef P 24b2931f243c419a9d1c7d00c97defac: Bootstrap starting.
I20260618 22:17:57.278700 2548 tablet_service.cc:1467] Processing CreateTablet for tablet d0114939b42040328f18490e90dd67ef (DEFAULT_TABLE table=test-table [id=359f690f3feb44d28ce647683e0c2cb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:17:57.278769 2732 tablet_bootstrap.cc:654] T d0114939b42040328f18490e90dd67ef P 24b2931f243c419a9d1c7d00c97defac: Neither blocks nor log segments found. Creating new log.
I20260618 22:17:57.278812 2731 tablet_bootstrap.cc:654] T d0114939b42040328f18490e90dd67ef P c5de8b6fc77c4d1090acf81f46f74192: Neither blocks nor log segments found. Creating new log.
I20260618 22:17:57.278963 2548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0114939b42040328f18490e90dd67ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:17:57.279883 2732 tablet_bootstrap.cc:492] T d0114939b42040328f18490e90dd67ef P 24b2931f243c419a9d1c7d00c97defac: No bootstrap required, opened a new log
I20260618 22:17:57.279891 2731 tablet_bootstrap.cc:492] T d0114939b42040328f18490e90dd67ef P c5de8b6fc77c4d1090acf81f46f74192: No bootstrap required, opened a new log
I20260618 22:17:57.279958 2731 ts_tablet_manager.cc:1397] T d0114939b42040328f18490e90dd67ef P c5de8b6fc77c4d1090acf81f46f74192: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260618 22:17:57.279959 2732 ts_tablet_manager.cc:1397] T d0114939b42040328f18490e90dd67ef P 24b2931f243c419a9d1c7d00c97defac: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260618 22:17:57.280380 2732 raft_consensus.cc:348] T d0114939b42040328f18490e90dd67ef P 24b2931f243c419a9d1c7d00c97defac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5de8b6fc77c4d1090acf81f46f74192" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35281 } } peers { permanent_uuid: "24b2931f243c419a9d1c7d00c97defac" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46229 } } peers { permanent_uuid: "58a33666811f42fca694ee4b30acb12f" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33011 } }
I20260618 22:17:57.280380 2731 raft_consensus.cc:348] T d0114939b42040328f18490e90dd67ef P c5de8b6fc77c4d1090acf81f46f74192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5de8b6fc77c4d1090acf81f46f74192" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35281 } } peers { permanent_uuid: "24b2931f243c419a9d1c7d00c97defac" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46229 } } peers { permanent_uuid: "58a33666811f42fca694ee4b30acb12f" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33011 } }
I20260618 22:17:57.280484 2732 raft_consensus.cc:374] T d0114939b42040328f18490e90dd67ef P 24b2931f243c419a9d1c7d00c97defac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:17:57.280508 2731 raft_consensus.cc:374] T d0114939b42040328f18490e90dd67ef P c5de8b6fc77c4d1090acf81f46f74192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:17:57.280519 2732 raft_consensus.cc:729] T d0114939b42040328f18490e90dd67ef P 24b2931f243c419a9d1c7d00c97defac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24b2931f243c419a9d1c7d00c97defac, State: Initialized, Role: FOLLOWER
I20260618 22:17:57.280546 2731 raft_consensus.cc:729] T d0114939b42040328f18490e90dd67ef P c5de8b6fc77c4d1090acf81f46f74192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5de8b6fc77c4d1090acf81f46f74192, State: Initialized, Role: FOLLOWER
I20260618 22:17:57.280652 2732 consensus_queue.cc:260] T d0114939b42040328f18490e90dd67ef P 24b2931f243c419a9d1c7d00c97defac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5de8b6fc77c4d1090acf81f46f74192" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35281 } } peers { permanent_uuid: "24b2931f243c419a9d1c7d00c97defac" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46229 } } peers { permanent_uuid: "58a33666811f42fca694ee4b30acb12f" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33011 } }
I20260618 22:17:57.280687 2731 consensus_queue.cc:260] T d0114939b42040328f18490e90dd67ef P c5de8b6fc77c4d1090acf81f46f74192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5de8b6fc77c4d1090acf81f46f74192" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35281 } } peers { permanent_uuid: "24b2931f243c419a9d1c7d00c97defac" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46229 } } peers { permanent_uuid: "58a33666811f42fca694ee4b30acb12f" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33011 } }
I20260618 22:17:57.280922 2732 ts_tablet_manager.cc:1428] T d0114939b42040328f18490e90dd67ef P 24b2931f243c419a9d1c7d00c97defac: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260618 22:17:57.281481 2731 ts_tablet_manager.cc:1428] T d0114939b42040328f18490e90dd67ef P c5de8b6fc77c4d1090acf81f46f74192: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260618 22:17:57.281693 2735 tablet_bootstrap.cc:492] T d0114939b42040328f18490e90dd67ef P 58a33666811f42fca694ee4b30acb12f: Bootstrap starting.
I20260618 22:17:57.283046 2735 tablet_bootstrap.cc:654] T d0114939b42040328f18490e90dd67ef P 58a33666811f42fca694ee4b30acb12f: Neither blocks nor log segments found. Creating new log.
I20260618 22:17:57.284479 2735 tablet_bootstrap.cc:492] T d0114939b42040328f18490e90dd67ef P 58a33666811f42fca694ee4b30acb12f: No bootstrap required, opened a new log
I20260618 22:17:57.284549 2735 ts_tablet_manager.cc:1397] T d0114939b42040328f18490e90dd67ef P 58a33666811f42fca694ee4b30acb12f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260618 22:17:57.284940 2735 raft_consensus.cc:348] T d0114939b42040328f18490e90dd67ef P 58a33666811f42fca694ee4b30acb12f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5de8b6fc77c4d1090acf81f46f74192" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35281 } } peers { permanent_uuid: "24b2931f243c419a9d1c7d00c97defac" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46229 } } peers { permanent_uuid: "58a33666811f42fca694ee4b30acb12f" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33011 } }
I20260618 22:17:57.285028 2735 raft_consensus.cc:374] T d0114939b42040328f18490e90dd67ef P 58a33666811f42fca694ee4b30acb12f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:17:57.285075 2735 raft_consensus.cc:729] T d0114939b42040328f18490e90dd67ef P 58a33666811f42fca694ee4b30acb12f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58a33666811f42fca694ee4b30acb12f, State: Initialized, Role: FOLLOWER
I20260618 22:17:57.285208 2735 consensus_queue.cc:260] T d0114939b42040328f18490e90dd67ef P 58a33666811f42fca694ee4b30acb12f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5de8b6fc77c4d1090acf81f46f74192" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35281 } } peers { permanent_uuid: "24b2931f243c419a9d1c7d00c97defac" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46229 } } peers { permanent_uuid: "58a33666811f42fca694ee4b30acb12f" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33011 } }
I20260618 22:17:57.285540 2735 ts_tablet_manager.cc:1428] T d0114939b42040328f18490e90dd67ef P 58a33666811f42fca694ee4b30acb12f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260618 22:17:57.330463 2737 raft_consensus.cc:482] T d0114939b42040328f18490e90dd67ef P 58a33666811f42fca694ee4b30acb12f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:17:57.330631 2737 raft_consensus.cc:504] T d0114939b42040328f18490e90dd67ef P 58a33666811f42fca694ee4b30acb12f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5de8b6fc77c4d1090acf81f46f74192" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35281 } } peers { permanent_uuid: "24b2931f243c419a9d1c7d00c97defac" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46229 } } peers { permanent_uuid: "58a33666811f42fca694ee4b30acb12f" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33011 } }
I20260618 22:17:57.331665 2737 leader_election.cc:288] T d0114939b42040328f18490e90dd67ef P 58a33666811f42fca694ee4b30acb12f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5de8b6fc77c4d1090acf81f46f74192 (127.0.98.65:35281), 24b2931f243c419a9d1c7d00c97defac (127.0.98.66:46229)
I20260618 22:17:57.336061 2738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:57.331909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:46229 (local address 127.0.98.67:50467)
0618 22:17:57.332282 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0618 22:17:57.332298 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 22:17:57.332402 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:17:57.332751 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:17:57.332757 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:17:57.332779 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:17:57.332907 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:57.332914 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:57.333714 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:57.333718 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:57.334802 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:57.334811 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:57.335550 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:57.335555 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:57.335653 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:57.335711 (+ 58us) client_negotiation.cc:769] Sending connection context
0618 22:17:57.335834 (+ 123us) client_negotiation.cc:240] Negotiation successful
0618 22:17:57.335903 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":93,"thread_start_us":119,"threads_started":1}
I20260618 22:17:57.336061 2739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:57.332010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:50467 (local address 127.0.98.66:46229)
0618 22:17:57.332351 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0618 22:17:57.332361 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0618 22:17:57.332381 (+ 20us) server_negotiation.cc:407] Connection header received
0618 22:17:57.332573 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:17:57.332579 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:17:57.332635 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:17:57.332719 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:17:57.333029 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:57.333599 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:57.334921 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:57.335428 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:57.335511 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:57.335593 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0618 22:17:57.335852 (+ 259us) server_negotiation.cc:299] Negotiation successful
0618 22:17:57.335923 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":87,"threads_started":1}
I20260618 22:17:57.336170 2740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:57.331909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:35281 (local address 127.0.98.67:33047)
0618 22:17:57.332350 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0618 22:17:57.332365 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:17:57.332530 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:17:57.332860 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:17:57.332863 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:17:57.332877 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:17:57.332994 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:57.333000 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:57.333778 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:57.333781 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:57.334878 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:57.334891 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:57.335657 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:57.335661 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:57.335783 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:57.335817 (+ 34us) client_negotiation.cc:769] Sending connection context
0618 22:17:57.335894 (+ 77us) client_negotiation.cc:240] Negotiation successful
0618 22:17:57.335959 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"mutex_wait_us":69,"thread_start_us":130,"threads_started":1}
I20260618 22:17:57.336488 2428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0114939b42040328f18490e90dd67ef" candidate_uuid: "58a33666811f42fca694ee4b30acb12f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24b2931f243c419a9d1c7d00c97defac" is_pre_election: true
I20260618 22:17:57.336573 2741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:57.332023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:33047 (local address 127.0.98.65:35281)
0618 22:17:57.332294 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0618 22:17:57.332304 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0618 22:17:57.332469 (+ 165us) server_negotiation.cc:407] Connection header received
0618 22:17:57.332651 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:17:57.332654 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:17:57.332721 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:17:57.332815 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:17:57.333114 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:57.333607 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:57.335007 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:57.335505 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:57.335611 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:57.335675 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0618 22:17:57.336365 (+ 690us) server_negotiation.cc:299] Negotiation successful
0618 22:17:57.336441 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":104,"threads_started":1}
I20260618 22:17:57.336647 2428 raft_consensus.cc:2454] T d0114939b42040328f18490e90dd67ef P 24b2931f243c419a9d1c7d00c97defac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58a33666811f42fca694ee4b30acb12f in term 0.
I20260618 22:17:57.336928 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0114939b42040328f18490e90dd67ef" candidate_uuid: "58a33666811f42fca694ee4b30acb12f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5de8b6fc77c4d1090acf81f46f74192" is_pre_election: true
I20260618 22:17:57.337042 2288 raft_consensus.cc:2454] T d0114939b42040328f18490e90dd67ef P c5de8b6fc77c4d1090acf81f46f74192 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58a33666811f42fca694ee4b30acb12f in term 0.
I20260618 22:17:57.337049 2499 leader_election.cc:302] T d0114939b42040328f18490e90dd67ef P 58a33666811f42fca694ee4b30acb12f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24b2931f243c419a9d1c7d00c97defac, 58a33666811f42fca694ee4b30acb12f; no voters:
I20260618 22:17:57.337352 2737 raft_consensus.cc:2781] T d0114939b42040328f18490e90dd67ef P 58a33666811f42fca694ee4b30acb12f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:17:57.337414 2737 raft_consensus.cc:482] T d0114939b42040328f18490e90dd67ef P 58a33666811f42fca694ee4b30acb12f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:17:57.337451 2737 raft_consensus.cc:3037] T d0114939b42040328f18490e90dd67ef P 58a33666811f42fca694ee4b30acb12f [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:17:57.338552 2737 raft_consensus.cc:504] T d0114939b42040328f18490e90dd67ef P 58a33666811f42fca694ee4b30acb12f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5de8b6fc77c4d1090acf81f46f74192" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35281 } } peers { permanent_uuid: "24b2931f243c419a9d1c7d00c97defac" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46229 } } peers { permanent_uuid: "58a33666811f42fca694ee4b30acb12f" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33011 } }
I20260618 22:17:57.338958 2737 leader_election.cc:288] T d0114939b42040328f18490e90dd67ef P 58a33666811f42fca694ee4b30acb12f [CANDIDATE]: Term 1 election: Requested vote from peers c5de8b6fc77c4d1090acf81f46f74192 (127.0.98.65:35281), 24b2931f243c419a9d1c7d00c97defac (127.0.98.66:46229)
I20260618 22:17:57.339263 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0114939b42040328f18490e90dd67ef" candidate_uuid: "58a33666811f42fca694ee4b30acb12f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5de8b6fc77c4d1090acf81f46f74192"
I20260618 22:17:57.339260 2428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0114939b42040328f18490e90dd67ef" candidate_uuid: "58a33666811f42fca694ee4b30acb12f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24b2931f243c419a9d1c7d00c97defac"
I20260618 22:17:57.339365 2428 raft_consensus.cc:3037] T d0114939b42040328f18490e90dd67ef P 24b2931f243c419a9d1c7d00c97defac [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:17:57.339365 2288 raft_consensus.cc:3037] T d0114939b42040328f18490e90dd67ef P c5de8b6fc77c4d1090acf81f46f74192 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:17:57.340433 2288 raft_consensus.cc:2454] T d0114939b42040328f18490e90dd67ef P c5de8b6fc77c4d1090acf81f46f74192 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58a33666811f42fca694ee4b30acb12f in term 1.
I20260618 22:17:57.340433 2428 raft_consensus.cc:2454] T d0114939b42040328f18490e90dd67ef P 24b2931f243c419a9d1c7d00c97defac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58a33666811f42fca694ee4b30acb12f in term 1.
I20260618 22:17:57.340699 2499 leader_election.cc:302] T d0114939b42040328f18490e90dd67ef P 58a33666811f42fca694ee4b30acb12f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24b2931f243c419a9d1c7d00c97defac, 58a33666811f42fca694ee4b30acb12f; no voters:
I20260618 22:17:57.340873 2737 raft_consensus.cc:2781] T d0114939b42040328f18490e90dd67ef P 58a33666811f42fca694ee4b30acb12f [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:17:57.341112 2737 raft_consensus.cc:686] T d0114939b42040328f18490e90dd67ef P 58a33666811f42fca694ee4b30acb12f [term 1 LEADER]: Becoming Leader. State: Replica: 58a33666811f42fca694ee4b30acb12f, State: Running, Role: LEADER
I20260618 22:17:57.341354 2737 consensus_queue.cc:237] T d0114939b42040328f18490e90dd67ef P 58a33666811f42fca694ee4b30acb12f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5de8b6fc77c4d1090acf81f46f74192" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35281 } } peers { permanent_uuid: "24b2931f243c419a9d1c7d00c97defac" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46229 } } peers { permanent_uuid: "58a33666811f42fca694ee4b30acb12f" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33011 } }
I20260618 22:17:57.343413 2131 catalog_manager.cc:5495] T d0114939b42040328f18490e90dd67ef P 58a33666811f42fca694ee4b30acb12f reported cstate change: term changed from 0 to 1, leader changed from <none> to 58a33666811f42fca694ee4b30acb12f (127.0.98.67). New cstate: current_term: 1 leader_uuid: "58a33666811f42fca694ee4b30acb12f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5de8b6fc77c4d1090acf81f46f74192" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24b2931f243c419a9d1c7d00c97defac" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58a33666811f42fca694ee4b30acb12f" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33011 } health_report { overall_health: HEALTHY } } }
I20260618 22:17:57.367666 2745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:57.363447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48368 (local address 127.0.98.67:33011)
0618 22:17:57.363747 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0618 22:17:57.363755 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0618 22:17:57.363765 (+ 10us) server_negotiation.cc:407] Connection header received
0618 22:17:57.363826 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:17:57.363828 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:17:57.363877 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:17:57.364005 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:17:57.364262 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:57.364807 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:57.365751 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:57.366111 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:57.366281 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:57.366558 (+ 277us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:17:57.367099 (+ 541us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:17:57.367318 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0618 22:17:57.367351 (+ 33us) server_negotiation.cc:299] Negotiation successful
0618 22:17:57.367399 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":94,"threads_started":1}
I20260618 22:17:57.377758 2739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:57.374483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:47659 (local address 127.0.98.66:46229)
0618 22:17:57.374612 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0618 22:17:57.374615 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:17:57.374628 (+ 13us) server_negotiation.cc:407] Connection header received
0618 22:17:57.374891 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:17:57.374894 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:17:57.374942 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:17:57.375015 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:17:57.375351 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:57.375832 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:57.376735 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:57.376958 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:57.377036 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:57.377393 (+ 357us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:17:57.377455 (+ 62us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:17:57.377512 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0618 22:17:57.377622 (+ 110us) server_negotiation.cc:299] Negotiation successful
0618 22:17:57.377662 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260618 22:17:57.377774 2748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:57.374265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:46229 (local address 127.0.98.67:47659)
0618 22:17:57.374577 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0618 22:17:57.374590 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:17:57.374728 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:17:57.375099 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:17:57.375102 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:17:57.375110 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0618 22:17:57.375236 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:57.375241 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:57.375971 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:57.375974 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:57.376614 (+ 640us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:57.376634 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:57.377077 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:57.377080 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:57.377150 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:57.377264 (+ 114us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0618 22:17:57.377537 (+ 273us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0618 22:17:57.377542 (+ 5us) client_negotiation.cc:769] Sending connection context
0618 22:17:57.377590 (+ 48us) client_negotiation.cc:240] Negotiation successful
0618 22:17:57.377650 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":127,"threads_started":1}
I20260618 22:17:57.383888 2428 raft_consensus.cc:1261] T d0114939b42040328f18490e90dd67ef P 24b2931f243c419a9d1c7d00c97defac [term 1 FOLLOWER]: Refusing update from remote peer 58a33666811f42fca694ee4b30acb12f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:17:57.383921 2288 raft_consensus.cc:1261] T d0114939b42040328f18490e90dd67ef P c5de8b6fc77c4d1090acf81f46f74192 [term 1 FOLLOWER]: Refusing update from remote peer 58a33666811f42fca694ee4b30acb12f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:17:57.384373 2742 consensus_queue.cc:1035] T d0114939b42040328f18490e90dd67ef P 58a33666811f42fca694ee4b30acb12f [LEADER]: Connected to new peer: Peer: permanent_uuid: "24b2931f243c419a9d1c7d00c97defac" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:17:57.384665 2737 consensus_queue.cc:1035] T d0114939b42040328f18490e90dd67ef P 58a33666811f42fca694ee4b30acb12f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5de8b6fc77c4d1090acf81f46f74192" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:17:57.387730 2702 mvcc.cc:205] Tried to move back new op lower bound from 7298339132960403456 to 7298339132792213504. Current Snapshot: MvccSnapshot[applied={T|T < 7298339132960403456}]
I20260618 22:17:57.401929 2753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:57.398354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:33011 (local address 127.0.98.66:48123)
0618 22:17:57.398639 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0618 22:17:57.398655 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 22:17:57.398748 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:17:57.398993 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:17:57.398996 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:17:57.399004 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0618 22:17:57.399107 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:57.399118 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:57.399858 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:57.399861 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:57.400520 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:17:57.400529 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:17:57.401121 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:17:57.401124 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:17:57.401242 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:57.401273 (+ 31us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0618 22:17:57.401603 (+ 330us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0618 22:17:57.401609 (+ 6us) client_negotiation.cc:769] Sending connection context
0618 22:17:57.401668 (+ 59us) client_negotiation.cc:240] Negotiation successful
0618 22:17:57.401762 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":88,"threads_started":1}
I20260618 22:17:57.401966 2745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:57.398453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:48123 (local address 127.0.98.67:33011)
0618 22:17:57.398564 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0618 22:17:57.398569 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:17:57.398737 (+ 168us) server_negotiation.cc:407] Connection header received
0618 22:17:57.398828 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:17:57.398831 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:17:57.398885 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:17:57.398968 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:17:57.399233 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:57.399728 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:57.400648 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:17:57.400967 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:17:57.401081 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:17:57.401421 (+ 340us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:17:57.401499 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:17:57.401578 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0618 22:17:57.401786 (+ 208us) server_negotiation.cc:299] Negotiation successful
0618 22:17:57.401832 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260618 22:17:57.424685 2131 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:17:57.424832 2131 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:17:57.427258 2131 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c640d89b20c44a6cb8af3a75a11e252a: Sending DeleteTablet for 3 replicas of tablet d0114939b42040328f18490e90dd67ef
I20260618 22:17:57.427948 2401 tablet_service.cc:1514] Processing DeleteTablet for tablet d0114939b42040328f18490e90dd67ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:17:57 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:40906
I20260618 22:17:57.427947 2268 tablet_service.cc:1514] Processing DeleteTablet for tablet d0114939b42040328f18490e90dd67ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:17:57 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:39364
I20260618 22:17:57.428133 2721 tablet_replica.cc:330] T d0114939b42040328f18490e90dd67ef P 24b2931f243c419a9d1c7d00c97defac: stopping tablet replica
I20260618 22:17:57.428243 2548 tablet_service.cc:1514] Processing DeleteTablet for tablet d0114939b42040328f18490e90dd67ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:17:57 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:48362
I20260618 22:17:57.428243 2720 tablet_replica.cc:330] T d0114939b42040328f18490e90dd67ef P c5de8b6fc77c4d1090acf81f46f74192: stopping tablet replica
I20260618 22:17:57.428289 2721 raft_consensus.cc:2229] T d0114939b42040328f18490e90dd67ef P 24b2931f243c419a9d1c7d00c97defac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:17:57.428388 2720 raft_consensus.cc:2229] T d0114939b42040328f18490e90dd67ef P c5de8b6fc77c4d1090acf81f46f74192 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:17:57.428406 2722 tablet_replica.cc:330] T d0114939b42040328f18490e90dd67ef P 58a33666811f42fca694ee4b30acb12f: stopping tablet replica
I20260618 22:17:57.428403 2721 raft_consensus.cc:2258] T d0114939b42040328f18490e90dd67ef P 24b2931f243c419a9d1c7d00c97defac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:17:57.428506 2720 raft_consensus.cc:2258] T d0114939b42040328f18490e90dd67ef P c5de8b6fc77c4d1090acf81f46f74192 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:17:57.428507 2722 raft_consensus.cc:2229] T d0114939b42040328f18490e90dd67ef P 58a33666811f42fca694ee4b30acb12f [term 1 LEADER]: Raft consensus shutting down.
I20260618 22:17:57.428817 2722 raft_consensus.cc:2258] T d0114939b42040328f18490e90dd67ef P 58a33666811f42fca694ee4b30acb12f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:17:57.429307 2720 ts_tablet_manager.cc:1905] T d0114939b42040328f18490e90dd67ef P c5de8b6fc77c4d1090acf81f46f74192: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:17:57.429579 2722 ts_tablet_manager.cc:1905] T d0114939b42040328f18490e90dd67ef P 58a33666811f42fca694ee4b30acb12f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:17:57.429567 2131 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38912:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 22:17:57.430017 2131 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:17:57.431464 2721 ts_tablet_manager.cc:1905] T d0114939b42040328f18490e90dd67ef P 24b2931f243c419a9d1c7d00c97defac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:17:57.434682 2268 tablet_service.cc:1467] Processing CreateTablet for tablet 20b9762321044c759bdb4076cc97840e (DEFAULT_TABLE table=test-table [id=31cce0eb1aff44b7bb8ec3747fe48ad5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:17:57.434828 2548 tablet_service.cc:1467] Processing CreateTablet for tablet 20b9762321044c759bdb4076cc97840e (DEFAULT_TABLE table=test-table [id=31cce0eb1aff44b7bb8ec3747fe48ad5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:17:57.434828 2401 tablet_service.cc:1467] Processing CreateTablet for tablet 20b9762321044c759bdb4076cc97840e (DEFAULT_TABLE table=test-table [id=31cce0eb1aff44b7bb8ec3747fe48ad5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:17:57.447391 2268 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20b9762321044c759bdb4076cc97840e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:17:57.447767 2401 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20b9762321044c759bdb4076cc97840e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:17:57.449819 2731 tablet_bootstrap.cc:492] T 20b9762321044c759bdb4076cc97840e P c5de8b6fc77c4d1090acf81f46f74192: Bootstrap starting.
I20260618 22:17:57.449939 2548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20b9762321044c759bdb4076cc97840e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:17:57.450702 2731 tablet_bootstrap.cc:654] T 20b9762321044c759bdb4076cc97840e P c5de8b6fc77c4d1090acf81f46f74192: Neither blocks nor log segments found. Creating new log.
I20260618 22:17:57.452330 2735 tablet_bootstrap.cc:492] T 20b9762321044c759bdb4076cc97840e P 58a33666811f42fca694ee4b30acb12f: Bootstrap starting.
I20260618 22:17:57.452529 2732 tablet_bootstrap.cc:492] T 20b9762321044c759bdb4076cc97840e P 24b2931f243c419a9d1c7d00c97defac: Bootstrap starting.
I20260618 22:17:57.453120 2731 tablet_bootstrap.cc:492] T 20b9762321044c759bdb4076cc97840e P c5de8b6fc77c4d1090acf81f46f74192: No bootstrap required, opened a new log
I20260618 22:17:57.453191 2731 ts_tablet_manager.cc:1397] T 20b9762321044c759bdb4076cc97840e P c5de8b6fc77c4d1090acf81f46f74192: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260618 22:17:57.453271 2735 tablet_bootstrap.cc:654] T 20b9762321044c759bdb4076cc97840e P 58a33666811f42fca694ee4b30acb12f: Neither blocks nor log segments found. Creating new log.
I20260618 22:17:57.453289 2732 tablet_bootstrap.cc:654] T 20b9762321044c759bdb4076cc97840e P 24b2931f243c419a9d1c7d00c97defac: Neither blocks nor log segments found. Creating new log.
I20260618 22:17:57.453612 2731 raft_consensus.cc:348] T 20b9762321044c759bdb4076cc97840e P c5de8b6fc77c4d1090acf81f46f74192 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5de8b6fc77c4d1090acf81f46f74192" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35281 } } peers { permanent_uuid: "24b2931f243c419a9d1c7d00c97defac" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46229 } } peers { permanent_uuid: "58a33666811f42fca694ee4b30acb12f" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33011 } }
I20260618 22:17:57.453739 2731 raft_consensus.cc:374] T 20b9762321044c759bdb4076cc97840e P c5de8b6fc77c4d1090acf81f46f74192 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:17:57.453770 2731 raft_consensus.cc:729] T 20b9762321044c759bdb4076cc97840e P c5de8b6fc77c4d1090acf81f46f74192 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5de8b6fc77c4d1090acf81f46f74192, State: Initialized, Role: FOLLOWER
I20260618 22:17:57.453917 2731 consensus_queue.cc:260] T 20b9762321044c759bdb4076cc97840e P c5de8b6fc77c4d1090acf81f46f74192 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5de8b6fc77c4d1090acf81f46f74192" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35281 } } peers { permanent_uuid: "24b2931f243c419a9d1c7d00c97defac" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46229 } } peers { permanent_uuid: "58a33666811f42fca694ee4b30acb12f" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33011 } }
I20260618 22:17:57.454182 2731 ts_tablet_manager.cc:1428] T 20b9762321044c759bdb4076cc97840e P c5de8b6fc77c4d1090acf81f46f74192: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260618 22:17:57.454963 2732 tablet_bootstrap.cc:492] T 20b9762321044c759bdb4076cc97840e P 24b2931f243c419a9d1c7d00c97defac: No bootstrap required, opened a new log
I20260618 22:17:57.454963 2735 tablet_bootstrap.cc:492] T 20b9762321044c759bdb4076cc97840e P 58a33666811f42fca694ee4b30acb12f: No bootstrap required, opened a new log
I20260618 22:17:57.455044 2732 ts_tablet_manager.cc:1397] T 20b9762321044c759bdb4076cc97840e P 24b2931f243c419a9d1c7d00c97defac: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260618 22:17:57.455044 2735 ts_tablet_manager.cc:1397] T 20b9762321044c759bdb4076cc97840e P 58a33666811f42fca694ee4b30acb12f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260618 22:17:57.455413 2732 raft_consensus.cc:348] T 20b9762321044c759bdb4076cc97840e P 24b2931f243c419a9d1c7d00c97defac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5de8b6fc77c4d1090acf81f46f74192" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35281 } } peers { permanent_uuid: "24b2931f243c419a9d1c7d00c97defac" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46229 } } peers { permanent_uuid: "58a33666811f42fca694ee4b30acb12f" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33011 } }
I20260618 22:17:57.455427 2735 raft_consensus.cc:348] T 20b9762321044c759bdb4076cc97840e P 58a33666811f42fca694ee4b30acb12f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5de8b6fc77c4d1090acf81f46f74192" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35281 } } peers { permanent_uuid: "24b2931f243c419a9d1c7d00c97defac" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46229 } } peers { permanent_uuid: "58a33666811f42fca694ee4b30acb12f" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33011 } }
I20260618 22:17:57.455516 2732 raft_consensus.cc:374] T 20b9762321044c759bdb4076cc97840e P 24b2931f243c419a9d1c7d00c97defac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:17:57.455516 2735 raft_consensus.cc:374] T 20b9762321044c759bdb4076cc97840e P 58a33666811f42fca694ee4b30acb12f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:17:57.455549 2735 raft_consensus.cc:729] T 20b9762321044c759bdb4076cc97840e P 58a33666811f42fca694ee4b30acb12f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58a33666811f42fca694ee4b30acb12f, State: Initialized, Role: FOLLOWER
I20260618 22:17:57.455549 2732 raft_consensus.cc:729] T 20b9762321044c759bdb4076cc97840e P 24b2931f243c419a9d1c7d00c97defac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24b2931f243c419a9d1c7d00c97defac, State: Initialized, Role: FOLLOWER
I20260618 22:17:57.455670 2735 consensus_queue.cc:260] T 20b9762321044c759bdb4076cc97840e P 58a33666811f42fca694ee4b30acb12f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5de8b6fc77c4d1090acf81f46f74192" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35281 } } peers { permanent_uuid: "24b2931f243c419a9d1c7d00c97defac" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46229 } } peers { permanent_uuid: "58a33666811f42fca694ee4b30acb12f" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33011 } }
I20260618 22:17:57.455672 2732 consensus_queue.cc:260] T 20b9762321044c759bdb4076cc97840e P 24b2931f243c419a9d1c7d00c97defac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5de8b6fc77c4d1090acf81f46f74192" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35281 } } peers { permanent_uuid: "24b2931f243c419a9d1c7d00c97defac" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46229 } } peers { permanent_uuid: "58a33666811f42fca694ee4b30acb12f" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33011 } }
I20260618 22:17:57.455920 2735 ts_tablet_manager.cc:1428] T 20b9762321044c759bdb4076cc97840e P 58a33666811f42fca694ee4b30acb12f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260618 22:17:57.455945 2732 ts_tablet_manager.cc:1428] T 20b9762321044c759bdb4076cc97840e P 24b2931f243c419a9d1c7d00c97defac: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260618 22:17:57.476578 2722 ts_tablet_manager.cc:1918] T d0114939b42040328f18490e90dd67ef P 58a33666811f42fca694ee4b30acb12f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260618 22:17:57.476578 2721 ts_tablet_manager.cc:1918] T d0114939b42040328f18490e90dd67ef P 24b2931f243c419a9d1c7d00c97defac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260618 22:17:57.476680 2722 log.cc:1196] T d0114939b42040328f18490e90dd67ef P 58a33666811f42fca694ee4b30acb12f: Deleting WAL directory at /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/ts-2/wal/wals/d0114939b42040328f18490e90dd67ef
I20260618 22:17:57.476688 2721 log.cc:1196] T d0114939b42040328f18490e90dd67ef P 24b2931f243c419a9d1c7d00c97defac: Deleting WAL directory at /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/ts-1/wal/wals/d0114939b42040328f18490e90dd67ef
I20260618 22:17:57.476960 2722 ts_tablet_manager.cc:1939] T d0114939b42040328f18490e90dd67ef P 58a33666811f42fca694ee4b30acb12f: Deleting consensus metadata
I20260618 22:17:57.476996 2721 ts_tablet_manager.cc:1939] T d0114939b42040328f18490e90dd67ef P 24b2931f243c419a9d1c7d00c97defac: Deleting consensus metadata
I20260618 22:17:57.477299 2720 ts_tablet_manager.cc:1918] T d0114939b42040328f18490e90dd67ef P c5de8b6fc77c4d1090acf81f46f74192: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260618 22:17:57.477389 2720 log.cc:1196] T d0114939b42040328f18490e90dd67ef P c5de8b6fc77c4d1090acf81f46f74192: Deleting WAL directory at /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/ts-0/wal/wals/d0114939b42040328f18490e90dd67ef
I20260618 22:17:57.477644 2720 ts_tablet_manager.cc:1939] T d0114939b42040328f18490e90dd67ef P c5de8b6fc77c4d1090acf81f46f74192: Deleting consensus metadata
I20260618 22:17:57.480712 2118 catalog_manager.cc:4841] TS 58a33666811f42fca694ee4b30acb12f (127.0.98.67:33011): tablet d0114939b42040328f18490e90dd67ef (table test-table [id=359f690f3feb44d28ce647683e0c2cb5]) successfully deleted
I20260618 22:17:57.480847 2116 catalog_manager.cc:4841] TS c5de8b6fc77c4d1090acf81f46f74192 (127.0.98.65:35281): tablet d0114939b42040328f18490e90dd67ef (table test-table [id=359f690f3feb44d28ce647683e0c2cb5]) successfully deleted
I20260618 22:17:57.480712 2115 catalog_manager.cc:4841] TS 24b2931f243c419a9d1c7d00c97defac (127.0.98.66:46229): tablet d0114939b42040328f18490e90dd67ef (table test-table [id=359f690f3feb44d28ce647683e0c2cb5]) successfully deleted
I20260618 22:17:57.785823 2751 raft_consensus.cc:482] T 20b9762321044c759bdb4076cc97840e P 58a33666811f42fca694ee4b30acb12f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:17:57.786008 2751 raft_consensus.cc:504] T 20b9762321044c759bdb4076cc97840e P 58a33666811f42fca694ee4b30acb12f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5de8b6fc77c4d1090acf81f46f74192" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35281 } } peers { permanent_uuid: "24b2931f243c419a9d1c7d00c97defac" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46229 } } peers { permanent_uuid: "58a33666811f42fca694ee4b30acb12f" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33011 } }
I20260618 22:17:57.786442 2751 leader_election.cc:288] T 20b9762321044c759bdb4076cc97840e P 58a33666811f42fca694ee4b30acb12f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c5de8b6fc77c4d1090acf81f46f74192 (127.0.98.65:35281), 24b2931f243c419a9d1c7d00c97defac (127.0.98.66:46229)
I20260618 22:17:57.787048 2428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20b9762321044c759bdb4076cc97840e" candidate_uuid: "58a33666811f42fca694ee4b30acb12f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24b2931f243c419a9d1c7d00c97defac" is_pre_election: true
I20260618 22:17:57.787061 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20b9762321044c759bdb4076cc97840e" candidate_uuid: "58a33666811f42fca694ee4b30acb12f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5de8b6fc77c4d1090acf81f46f74192" is_pre_election: true
I20260618 22:17:57.787205 2428 raft_consensus.cc:2454] T 20b9762321044c759bdb4076cc97840e P 24b2931f243c419a9d1c7d00c97defac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58a33666811f42fca694ee4b30acb12f in term 0.
I20260618 22:17:57.787232 2288 raft_consensus.cc:2454] T 20b9762321044c759bdb4076cc97840e P c5de8b6fc77c4d1090acf81f46f74192 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 58a33666811f42fca694ee4b30acb12f in term 0.
I20260618 22:17:57.787511 2499 leader_election.cc:302] T 20b9762321044c759bdb4076cc97840e P 58a33666811f42fca694ee4b30acb12f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24b2931f243c419a9d1c7d00c97defac, 58a33666811f42fca694ee4b30acb12f; no voters:
I20260618 22:17:57.787705 2751 raft_consensus.cc:2781] T 20b9762321044c759bdb4076cc97840e P 58a33666811f42fca694ee4b30acb12f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:17:57.787770 2751 raft_consensus.cc:482] T 20b9762321044c759bdb4076cc97840e P 58a33666811f42fca694ee4b30acb12f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:17:57.787801 2751 raft_consensus.cc:3037] T 20b9762321044c759bdb4076cc97840e P 58a33666811f42fca694ee4b30acb12f [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:17:57.788779 2751 raft_consensus.cc:504] T 20b9762321044c759bdb4076cc97840e P 58a33666811f42fca694ee4b30acb12f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5de8b6fc77c4d1090acf81f46f74192" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35281 } } peers { permanent_uuid: "24b2931f243c419a9d1c7d00c97defac" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46229 } } peers { permanent_uuid: "58a33666811f42fca694ee4b30acb12f" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33011 } }
I20260618 22:17:57.789142 2751 leader_election.cc:288] T 20b9762321044c759bdb4076cc97840e P 58a33666811f42fca694ee4b30acb12f [CANDIDATE]: Term 1 election: Requested vote from peers c5de8b6fc77c4d1090acf81f46f74192 (127.0.98.65:35281), 24b2931f243c419a9d1c7d00c97defac (127.0.98.66:46229)
I20260618 22:17:57.789494 2428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20b9762321044c759bdb4076cc97840e" candidate_uuid: "58a33666811f42fca694ee4b30acb12f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24b2931f243c419a9d1c7d00c97defac"
I20260618 22:17:57.789626 2428 raft_consensus.cc:3037] T 20b9762321044c759bdb4076cc97840e P 24b2931f243c419a9d1c7d00c97defac [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:17:57.789578 2288 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20b9762321044c759bdb4076cc97840e" candidate_uuid: "58a33666811f42fca694ee4b30acb12f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c5de8b6fc77c4d1090acf81f46f74192"
I20260618 22:17:57.789748 2288 raft_consensus.cc:3037] T 20b9762321044c759bdb4076cc97840e P c5de8b6fc77c4d1090acf81f46f74192 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:17:57.790686 2428 raft_consensus.cc:2454] T 20b9762321044c759bdb4076cc97840e P 24b2931f243c419a9d1c7d00c97defac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58a33666811f42fca694ee4b30acb12f in term 1.
I20260618 22:17:57.790876 2288 raft_consensus.cc:2454] T 20b9762321044c759bdb4076cc97840e P c5de8b6fc77c4d1090acf81f46f74192 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 58a33666811f42fca694ee4b30acb12f in term 1.
I20260618 22:17:57.791010 2499 leader_election.cc:302] T 20b9762321044c759bdb4076cc97840e P 58a33666811f42fca694ee4b30acb12f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24b2931f243c419a9d1c7d00c97defac, 58a33666811f42fca694ee4b30acb12f; no voters:
I20260618 22:17:57.791193 2751 raft_consensus.cc:2781] T 20b9762321044c759bdb4076cc97840e P 58a33666811f42fca694ee4b30acb12f [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:17:57.791320 2751 raft_consensus.cc:686] T 20b9762321044c759bdb4076cc97840e P 58a33666811f42fca694ee4b30acb12f [term 1 LEADER]: Becoming Leader. State: Replica: 58a33666811f42fca694ee4b30acb12f, State: Running, Role: LEADER
I20260618 22:17:57.791456 2751 consensus_queue.cc:237] T 20b9762321044c759bdb4076cc97840e P 58a33666811f42fca694ee4b30acb12f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5de8b6fc77c4d1090acf81f46f74192" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35281 } } peers { permanent_uuid: "24b2931f243c419a9d1c7d00c97defac" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46229 } } peers { permanent_uuid: "58a33666811f42fca694ee4b30acb12f" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33011 } }
I20260618 22:17:57.793020 2131 catalog_manager.cc:5495] T 20b9762321044c759bdb4076cc97840e P 58a33666811f42fca694ee4b30acb12f reported cstate change: term changed from 0 to 1, leader changed from <none> to 58a33666811f42fca694ee4b30acb12f (127.0.98.67). New cstate: current_term: 1 leader_uuid: "58a33666811f42fca694ee4b30acb12f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5de8b6fc77c4d1090acf81f46f74192" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "24b2931f243c419a9d1c7d00c97defac" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46229 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "58a33666811f42fca694ee4b30acb12f" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33011 } health_report { overall_health: HEALTHY } } }
I20260618 22:17:57.825098 2428 raft_consensus.cc:1261] T 20b9762321044c759bdb4076cc97840e P 24b2931f243c419a9d1c7d00c97defac [term 1 FOLLOWER]: Refusing update from remote peer 58a33666811f42fca694ee4b30acb12f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:17:57.825165 2288 raft_consensus.cc:1261] T 20b9762321044c759bdb4076cc97840e P c5de8b6fc77c4d1090acf81f46f74192 [term 1 FOLLOWER]: Refusing update from remote peer 58a33666811f42fca694ee4b30acb12f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:17:57.825502 2751 consensus_queue.cc:1035] T 20b9762321044c759bdb4076cc97840e P 58a33666811f42fca694ee4b30acb12f [LEADER]: Connected to new peer: Peer: permanent_uuid: "24b2931f243c419a9d1c7d00c97defac" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46229 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:17:57.825702 2742 consensus_queue.cc:1035] T 20b9762321044c759bdb4076cc97840e P 58a33666811f42fca694ee4b30acb12f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5de8b6fc77c4d1090acf81f46f74192" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 35281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260618 22:17:57.838022 2702 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260618 22:17:59.385637 2130 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:17:59.385812 2130 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:17:59.388002 2130 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c640d89b20c44a6cb8af3a75a11e252a: Sending DeleteTablet for 3 replicas of tablet 20b9762321044c759bdb4076cc97840e
I20260618 22:17:59.388533 2268 tablet_service.cc:1514] Processing DeleteTablet for tablet 20b9762321044c759bdb4076cc97840e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:17:59 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:39364
I20260618 22:17:59.388572 2401 tablet_service.cc:1514] Processing DeleteTablet for tablet 20b9762321044c759bdb4076cc97840e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:17:59 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:40906
I20260618 22:17:59.388731 2548 tablet_service.cc:1514] Processing DeleteTablet for tablet 20b9762321044c759bdb4076cc97840e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:17:59 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:48362
I20260618 22:17:59.388852 2784 tablet_replica.cc:330] T 20b9762321044c759bdb4076cc97840e P 24b2931f243c419a9d1c7d00c97defac: stopping tablet replica
I20260618 22:17:59.388962 2785 tablet_replica.cc:330] T 20b9762321044c759bdb4076cc97840e P c5de8b6fc77c4d1090acf81f46f74192: stopping tablet replica
I20260618 22:17:59.389016 2786 tablet_replica.cc:330] T 20b9762321044c759bdb4076cc97840e P 58a33666811f42fca694ee4b30acb12f: stopping tablet replica
I20260618 22:17:59.389025 2784 raft_consensus.cc:2229] T 20b9762321044c759bdb4076cc97840e P 24b2931f243c419a9d1c7d00c97defac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:17:59.389035 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 2203
I20260618 22:17:59.389765 2786 raft_consensus.cc:2229] T 20b9762321044c759bdb4076cc97840e P 58a33666811f42fca694ee4b30acb12f [term 1 LEADER]: Raft consensus shutting down.
I20260618 22:17:59.389839 2784 raft_consensus.cc:2258] T 20b9762321044c759bdb4076cc97840e P 24b2931f243c419a9d1c7d00c97defac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:17:59.390044 2786 raft_consensus.cc:2258] T 20b9762321044c759bdb4076cc97840e P 58a33666811f42fca694ee4b30acb12f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:17:59.390713 2786 ts_tablet_manager.cc:1905] T 20b9762321044c759bdb4076cc97840e P 58a33666811f42fca694ee4b30acb12f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:17:59.390717 2784 ts_tablet_manager.cc:1905] T 20b9762321044c759bdb4076cc97840e P 24b2931f243c419a9d1c7d00c97defac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:17:59.393189 2786 ts_tablet_manager.cc:1918] T 20b9762321044c759bdb4076cc97840e P 58a33666811f42fca694ee4b30acb12f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260618 22:17:59.393290 2786 log.cc:1196] T 20b9762321044c759bdb4076cc97840e P 58a33666811f42fca694ee4b30acb12f: Deleting WAL directory at /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/ts-2/wal/wals/20b9762321044c759bdb4076cc97840e
I20260618 22:17:59.393566 2786 ts_tablet_manager.cc:1939] T 20b9762321044c759bdb4076cc97840e P 58a33666811f42fca694ee4b30acb12f: Deleting consensus metadata
I20260618 22:17:59.394064 2784 ts_tablet_manager.cc:1918] T 20b9762321044c759bdb4076cc97840e P 24b2931f243c419a9d1c7d00c97defac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260618 22:17:59.394142 2784 log.cc:1196] T 20b9762321044c759bdb4076cc97840e P 24b2931f243c419a9d1c7d00c97defac: Deleting WAL directory at /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781821065380026-393-0/minicluster-data/ts-1/wal/wals/20b9762321044c759bdb4076cc97840e
I20260618 22:17:59.394261 2118 catalog_manager.cc:4841] TS 58a33666811f42fca694ee4b30acb12f (127.0.98.67:33011): tablet 20b9762321044c759bdb4076cc97840e (table test-table [id=31cce0eb1aff44b7bb8ec3747fe48ad5]) successfully deleted
I20260618 22:17:59.394415 2784 ts_tablet_manager.cc:1939] T 20b9762321044c759bdb4076cc97840e P 24b2931f243c419a9d1c7d00c97defac: Deleting consensus metadata
I20260618 22:17:59.395117 2115 catalog_manager.cc:4841] TS 24b2931f243c419a9d1c7d00c97defac (127.0.98.66:46229): tablet 20b9762321044c759bdb4076cc97840e (table test-table [id=31cce0eb1aff44b7bb8ec3747fe48ad5]) successfully deleted
W20260618 22:17:59.397303 2116 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.65:35281 (error 108)
I20260618 22:17:59.397522 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 2343
I20260618 22:17:59.398904 2788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:17:59.398467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:35281 (local address 127.0.0.1:39382)
0618 22:17:59.398769 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0618 22:17:59.398830 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.65:35281: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":75,"threads_started":1}
W20260618 22:17:59.399134 2116 catalog_manager.cc:4584] TS c5de8b6fc77c4d1090acf81f46f74192 (127.0.98.65:35281): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 20b9762321044c759bdb4076cc97840e: Network error: Client connection negotiation failed: client connection to 127.0.98.65:35281: connect: Connection refused (error 111)
I20260618 22:17:59.404604 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 2483
I20260618 22:17:59.410864 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 2099
2026-06-18T22:17:59Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6451 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:17:59 dist-test-slave-g6r4 krb5kdc[2791](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:17:59 dist-test-slave-g6r4 krb5kdc[2791](info): set up 2 sockets
Jun 18 22:17:59 dist-test-slave-g6r4 krb5kdc[2791](info): commencing operation
krb5kdc: starting...
W20260618 22:18:00.453728 393 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/krb5kdc/test-user.keytab.
Jun 18 22:18:00 dist-test-slave-g6r4 krb5kdc[2791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:18:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:18:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260618 22:18:00.572656 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:46699
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:46085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:46699
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:18:00.679014 2804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:18:00.679246 2804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:18:00.679283 2804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:18:00.683277 2804 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:18:00.683329 2804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:18:00.683348 2804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:18:00.683364 2804 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:18:00.683379 2804 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:18:00.688470 2804 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:46085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:46699
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:46699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:18:00.689435 2804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:18:00.690474 2804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:18:00.698398 2812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:00.698479 2809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:00.698671 2804 server_base.cc:1034] running on GCE node
W20260618 22:18:00.698701 2810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:00.699208 2804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:18:00.700341 2804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:18:00.701560 2804 hybrid_clock.cc:648] HybridClock initialized: now 1781821080701520 us; error 47 us; skew 500 ppm
Jun 18 22:18:00 dist-test-slave-g6r4 krb5kdc[2791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821080, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:18:00.704342 2804 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260618 22:18:00.705247 2804 webserver.cc:460] Webserver started at http://127.0.98.126:35437/ using document root <none> and password file <none>
I20260618 22:18:00.705794 2804 fs_manager.cc:359] Metadata directory not provided
I20260618 22:18:00.705891 2804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:18:00.706115 2804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:18:00.708133 2804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/minicluster-data/master-0/data/instance:
uuid: "e969635594af43bcae2c0d90e1123afc"
format_stamp: "Formatted at 2026-06-18 22:18:00 on dist-test-slave-g6r4"
server_key: "dd3064862f8bd6af8da469d45b605fcb"
server_key_iv: "289661d5dc5d5e54792a1e04f669b2e2"
server_key_version: "encryptionkey@0"
I20260618 22:18:00.708676 2804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/minicluster-data/master-0/wal/instance:
uuid: "e969635594af43bcae2c0d90e1123afc"
format_stamp: "Formatted at 2026-06-18 22:18:00 on dist-test-slave-g6r4"
server_key: "dd3064862f8bd6af8da469d45b605fcb"
server_key_iv: "289661d5dc5d5e54792a1e04f669b2e2"
server_key_version: "encryptionkey@0"
I20260618 22:18:00.712225 2804 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260618 22:18:00.714547 2819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:00.715734 2804 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:18:00.715885 2804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/minicluster-data/master-0/wal
uuid: "e969635594af43bcae2c0d90e1123afc"
format_stamp: "Formatted at 2026-06-18 22:18:00 on dist-test-slave-g6r4"
server_key: "dd3064862f8bd6af8da469d45b605fcb"
server_key_iv: "289661d5dc5d5e54792a1e04f669b2e2"
server_key_version: "encryptionkey@0"
I20260618 22:18:00.715997 2804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:18:00.731637 2804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:18:00.734606 2804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:18:00.734815 2804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:18:00.741959 2871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:46699 every 8 connection(s)
I20260618 22:18:00.741966 2804 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:46699
I20260618 22:18:00.742935 2804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
I20260618 22:18:00.745530 2872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:18:00.748211 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 2804
I20260618 22:18:00.748322 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/minicluster-data/master-0/wal/instance
I20260618 22:18:00.748677 393 external_mini_cluster.cc:1428] Setting key f71a4eac05a1fc85a78e43fe714a75e1
I20260618 22:18:00.751308 2872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e969635594af43bcae2c0d90e1123afc: Bootstrap starting.
I20260618 22:18:00.753705 2872 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e969635594af43bcae2c0d90e1123afc: Neither blocks nor log segments found. Creating new log.
I20260618 22:18:00.754603 2872 log.cc:824] T 00000000000000000000000000000000 P e969635594af43bcae2c0d90e1123afc: Log is configured to *not* fsync() on all Append() calls
I20260618 22:18:00.756649 2872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e969635594af43bcae2c0d90e1123afc: No bootstrap required, opened a new log
Jun 18 22:18:00 dist-test-slave-g6r4 krb5kdc[2791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:18:00.760463 2872 raft_consensus.cc:348] T 00000000000000000000000000000000 P e969635594af43bcae2c0d90e1123afc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e969635594af43bcae2c0d90e1123afc" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46699 } }
I20260618 22:18:00.760701 2872 raft_consensus.cc:374] T 00000000000000000000000000000000 P e969635594af43bcae2c0d90e1123afc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:18:00.760766 2872 raft_consensus.cc:729] T 00000000000000000000000000000000 P e969635594af43bcae2c0d90e1123afc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e969635594af43bcae2c0d90e1123afc, State: Initialized, Role: FOLLOWER
I20260618 22:18:00.761226 2872 consensus_queue.cc:260] T 00000000000000000000000000000000 P e969635594af43bcae2c0d90e1123afc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e969635594af43bcae2c0d90e1123afc" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46699 } }
I20260618 22:18:00.761356 2872 raft_consensus.cc:388] T 00000000000000000000000000000000 P e969635594af43bcae2c0d90e1123afc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:18:00.761439 2872 raft_consensus.cc:482] T 00000000000000000000000000000000 P e969635594af43bcae2c0d90e1123afc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:18:00.761559 2872 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e969635594af43bcae2c0d90e1123afc [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:18:00.762485 2872 raft_consensus.cc:504] T 00000000000000000000000000000000 P e969635594af43bcae2c0d90e1123afc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e969635594af43bcae2c0d90e1123afc" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46699 } }
I20260618 22:18:00.762812 2872 leader_election.cc:302] T 00000000000000000000000000000000 P e969635594af43bcae2c0d90e1123afc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e969635594af43bcae2c0d90e1123afc; no voters:
I20260618 22:18:00.763108 2872 leader_election.cc:288] T 00000000000000000000000000000000 P e969635594af43bcae2c0d90e1123afc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:18:00.763186 2877 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e969635594af43bcae2c0d90e1123afc [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:18:00.763392 2877 raft_consensus.cc:686] T 00000000000000000000000000000000 P e969635594af43bcae2c0d90e1123afc [term 1 LEADER]: Becoming Leader. State: Replica: e969635594af43bcae2c0d90e1123afc, State: Running, Role: LEADER
I20260618 22:18:00.763515 2875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:00.750152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60876 (local address 127.0.98.126:46699)
0618 22:18:00.750538 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:00.750564 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:00.750598 (+ 34us) server_negotiation.cc:407] Connection header received
0618 22:18:00.751182 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:00.751197 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:00.751574 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:00.751859 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:00.752757 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:00.753777 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:00.754985 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:00.755582 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:00.755700 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:00.758435 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:00.758440 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:00.758452 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:00.758476 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:00.760230 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:00.760721 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:00.760734 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:00.760738 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:00.760923 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:00.761218 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:00.761222 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:00.761223 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:00.761761 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:00.762002 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:00.762285 (+ 283us) server_negotiation.cc:299] Negotiation successful
0618 22:18:00.762555 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":146,"threads_started":1}
I20260618 22:18:00.763691 2877 consensus_queue.cc:237] T 00000000000000000000000000000000 P e969635594af43bcae2c0d90e1123afc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e969635594af43bcae2c0d90e1123afc" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46699 } }
I20260618 22:18:00.764247 2872 sys_catalog.cc:564] T 00000000000000000000000000000000 P e969635594af43bcae2c0d90e1123afc [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:18:00.764736 2879 sys_catalog.cc:455] T 00000000000000000000000000000000 P e969635594af43bcae2c0d90e1123afc [sys.catalog]: SysCatalogTable state changed. Reason: New leader e969635594af43bcae2c0d90e1123afc. Latest consensus state: current_term: 1 leader_uuid: "e969635594af43bcae2c0d90e1123afc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e969635594af43bcae2c0d90e1123afc" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46699 } } }
I20260618 22:18:00.764881 2879 sys_catalog.cc:458] T 00000000000000000000000000000000 P e969635594af43bcae2c0d90e1123afc [sys.catalog]: This master's current role is: LEADER
I20260618 22:18:00.765579 2883 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:18:00.766764 2878 sys_catalog.cc:455] T 00000000000000000000000000000000 P e969635594af43bcae2c0d90e1123afc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e969635594af43bcae2c0d90e1123afc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e969635594af43bcae2c0d90e1123afc" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46699 } } }
I20260618 22:18:00.766866 2878 sys_catalog.cc:458] T 00000000000000000000000000000000 P e969635594af43bcae2c0d90e1123afc [sys.catalog]: This master's current role is: LEADER
I20260618 22:18:00.768556 2883 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:18:00.773684 2883 catalog_manager.cc:1347] Generated new cluster ID: 87c615639ce749f59d45dd39d5aa36be
I20260618 22:18:00.773782 2883 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:18:00.783023 2883 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:18:00.784062 2883 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:18:00.793211 2883 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e969635594af43bcae2c0d90e1123afc: Generated new TSK 0
I20260618 22:18:00.793874 2883 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab.
I20260618 22:18:00.829984 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:46699
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:46085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:18:00.970762 2898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:18:00.970988 2898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:18:00.971028 2898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:18:00.974962 2898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:18:00.975008 2898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:18:00.975063 2898 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260618 22:18:00.980021 2898 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:46085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:46699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2898
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:18:00.980974 2898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:18:00.981966 2898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:18:00.990792 2904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:00.990797 2903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:00.990808 2906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:00.991043 2898 server_base.cc:1034] running on GCE node
I20260618 22:18:00.991554 2898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:18:00.992151 2898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:18:00.993372 2898 hybrid_clock.cc:648] HybridClock initialized: now 1781821080993357 us; error 26 us; skew 500 ppm
Jun 18 22:18:00 dist-test-slave-g6r4 krb5kdc[2791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821080, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:18:00.996225 2898 init.cc:377] Logged in from keytab as kudu/127.0.98.65@KRBTEST.COM (short username kudu)
I20260618 22:18:00.997504 2898 webserver.cc:460] Webserver started at http://127.0.98.65:43347/ using document root <none> and password file <none>
I20260618 22:18:00.998127 2898 fs_manager.cc:359] Metadata directory not provided
I20260618 22:18:00.998224 2898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:18:00.998452 2898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:18:01.000380 2898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-0/data/instance:
uuid: "1b6370a9e24e4900872ec4d005e6f5f4"
format_stamp: "Formatted at 2026-06-18 22:18:00 on dist-test-slave-g6r4"
server_key: "b906a391edc1c0ab342eca3249fcfc28"
server_key_iv: "dbfeb5293583aad054cd95e65e8e9e63"
server_key_version: "encryptionkey@0"
I20260618 22:18:01.000952 2898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-0/wal/instance:
uuid: "1b6370a9e24e4900872ec4d005e6f5f4"
format_stamp: "Formatted at 2026-06-18 22:18:00 on dist-test-slave-g6r4"
server_key: "b906a391edc1c0ab342eca3249fcfc28"
server_key_iv: "dbfeb5293583aad054cd95e65e8e9e63"
server_key_version: "encryptionkey@0"
I20260618 22:18:01.004523 2898 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260618 22:18:01.007025 2913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:01.008205 2898 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:18:01.008347 2898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-0/wal
uuid: "1b6370a9e24e4900872ec4d005e6f5f4"
format_stamp: "Formatted at 2026-06-18 22:18:00 on dist-test-slave-g6r4"
server_key: "b906a391edc1c0ab342eca3249fcfc28"
server_key_iv: "dbfeb5293583aad054cd95e65e8e9e63"
server_key_version: "encryptionkey@0"
I20260618 22:18:01.008457 2898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:18:01.032522 2898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:18:01.035369 2898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:18:01.035575 2898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:18:01.036149 2898 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:18:01.036937 2898 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:18:01.037015 2898 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:01.037096 2898 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:18:01.037134 2898 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:01.046432 2898 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:41559
I20260618 22:18:01.046456 3026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:41559 every 8 connection(s)
I20260618 22:18:01.047267 2898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
I20260618 22:18:01.048120 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 2898
I20260618 22:18:01.048231 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-0/wal/instance
I20260618 22:18:01.048485 393 external_mini_cluster.cc:1428] Setting key 932c89bbc7ebea811e04e01863d6d602
WARNING: no policy specified for kudu/127.0.98.66@KRBTEST.COM; defaulting to no policy
Jun 18 22:18:01 dist-test-slave-g6r4 krb5kdc[2791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821080, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.66@KRBTEST.COM" created.
I20260618 22:18:01.062671 2875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:01.049096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:35161 (local address 127.0.98.126:46699)
0618 22:18:01.049223 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:01.049227 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:01.050225 (+ 998us) server_negotiation.cc:407] Connection header received
0618 22:18:01.052130 (+ 1905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:01.052136 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:01.052196 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:01.052294 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:01.053450 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:01.054384 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:01.055566 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:01.055979 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:01.056100 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:01.058581 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:01.058584 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:01.058586 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:01.058609 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:01.060232 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:01.060861 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:01.060867 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:01.060872 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:01.060963 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:01.061408 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:01.061410 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:01.061411 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:01.061567 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:01.061641 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:01.062318 (+ 677us) server_negotiation.cc:299] Negotiation successful
0618 22:18:01.062457 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260618 22:18:01.063295 3029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:01.049508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:46699 (local address 127.0.98.65:35161)
0618 22:18:01.050062 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:01.050119 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:01.051936 (+ 1817us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:01.052442 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:01.052448 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:01.052847 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:01.053300 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:01.053313 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:01.054525 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:01.054529 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:01.055432 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:01.055441 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:01.056153 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:01.056156 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:01.056296 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:01.056970 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:01.056985 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:01.058432 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:18:01.060359 (+ 1927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:01.060363 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:01.060374 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:01.060735 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:01.061086 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:01.061091 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:01.061096 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:01.061307 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:01.061676 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:18:01.061686 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:18:01.062014 (+ 328us) client_negotiation.cc:769] Sending connection context
0618 22:18:01.062250 (+ 236us) client_negotiation.cc:240] Negotiation successful
0618 22:18:01.062483 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":143,"threads_started":1}
I20260618 22:18:01.064543 3027 heartbeater.cc:347] Connected to a master server at 127.0.98.126:46699
I20260618 22:18:01.064831 3027 heartbeater.cc:464] Registering TS with master...
I20260618 22:18:01.065356 3027 heartbeater.cc:511] Master 127.0.98.126:46699 requested a full tablet report, sending...
I20260618 22:18:01.066807 2836 ts_manager.cc:194] Registered new tserver with Master: 1b6370a9e24e4900872ec4d005e6f5f4 (127.0.98.65:41559)
I20260618 22:18:01.067867 2836 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.65@KRBTEST.COM'} at 127.0.98.65:35161
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab.
I20260618 22:18:01.077153 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:46699
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:46085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:18:01.181885 3032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:18:01.182175 3032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:18:01.182250 3032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:18:01.186272 3032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:18:01.186367 3032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:18:01.186457 3032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260618 22:18:01.191483 3032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:46085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:46699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3032
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:18:01.192449 3032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:18:01.193419 3032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:18:01.202695 3037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:01.202746 3038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:01.202746 3040 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:01.203164 3032 server_base.cc:1034] running on GCE node
I20260618 22:18:01.203528 3032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:18:01.204154 3032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:18:01.205343 3032 hybrid_clock.cc:648] HybridClock initialized: now 1781821081205322 us; error 31 us; skew 500 ppm
Jun 18 22:18:01 dist-test-slave-g6r4 krb5kdc[2791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821081, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:18:01.208199 3032 init.cc:377] Logged in from keytab as kudu/127.0.98.66@KRBTEST.COM (short username kudu)
I20260618 22:18:01.209056 3032 webserver.cc:460] Webserver started at http://127.0.98.66:35163/ using document root <none> and password file <none>
I20260618 22:18:01.209620 3032 fs_manager.cc:359] Metadata directory not provided
I20260618 22:18:01.209719 3032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:18:01.210011 3032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:18:01.212088 3032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-1/data/instance:
uuid: "b752c702652048d39aedd8a1fdc5e475"
format_stamp: "Formatted at 2026-06-18 22:18:01 on dist-test-slave-g6r4"
server_key: "3792d466dae9dadbeda6d155586ff3ed"
server_key_iv: "90576d15638c18864783d901f9b0f12c"
server_key_version: "encryptionkey@0"
I20260618 22:18:01.212713 3032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-1/wal/instance:
uuid: "b752c702652048d39aedd8a1fdc5e475"
format_stamp: "Formatted at 2026-06-18 22:18:01 on dist-test-slave-g6r4"
server_key: "3792d466dae9dadbeda6d155586ff3ed"
server_key_iv: "90576d15638c18864783d901f9b0f12c"
server_key_version: "encryptionkey@0"
I20260618 22:18:01.216280 3032 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260618 22:18:01.218900 3047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:01.220302 3032 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:18:01.220459 3032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-1/wal
uuid: "b752c702652048d39aedd8a1fdc5e475"
format_stamp: "Formatted at 2026-06-18 22:18:01 on dist-test-slave-g6r4"
server_key: "3792d466dae9dadbeda6d155586ff3ed"
server_key_iv: "90576d15638c18864783d901f9b0f12c"
server_key_version: "encryptionkey@0"
I20260618 22:18:01.220582 3032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:18:01.248639 3032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:18:01.251942 3032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:18:01.252167 3032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:18:01.252852 3032 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:18:01.253680 3032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:18:01.253754 3032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:01.253826 3032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:18:01.253867 3032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:01.262784 3032 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:40183
I20260618 22:18:01.262809 3160 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:40183 every 8 connection(s)
I20260618 22:18:01.263665 3032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
I20260618 22:18:01.272821 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 3032
I20260618 22:18:01.272936 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-1/wal/instance
I20260618 22:18:01.273177 393 external_mini_cluster.cc:1428] Setting key 1db8fe4cf0c3f0f1c78cfb7f7245d9c7
Jun 18 22:18:01 dist-test-slave-g6r4 krb5kdc[2791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821081, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:18:01.278043 2875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:01.265611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:48241 (local address 127.0.98.126:46699)
0618 22:18:01.265754 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:01.265758 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:01.266623 (+ 865us) server_negotiation.cc:407] Connection header received
0618 22:18:01.268052 (+ 1429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:01.268055 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:01.268108 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:01.268190 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:01.269423 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:01.270022 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:01.271245 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:01.271479 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:01.271590 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:01.274254 (+ 2664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:01.274257 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:01.274260 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:01.274281 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:01.275851 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:01.276417 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:01.276423 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:01.276425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:01.276512 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:01.276890 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:01.276895 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:01.276897 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:01.277084 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:01.277217 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:01.277712 (+ 495us) server_negotiation.cc:299] Negotiation successful
0618 22:18:01.277832 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260618 22:18:01.278791 3163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:01.266032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:46699 (local address 127.0.98.66:48241)
0618 22:18:01.266478 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:01.266517 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:01.267846 (+ 1329us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:01.268367 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:01.268373 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:01.268715 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:01.269280 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:01.269294 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:01.270154 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:01.270157 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:01.271128 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:01.271137 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:01.271621 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:01.271625 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:01.271782 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:01.272306 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:01.272320 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:01.274103 (+ 1783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:18:01.275989 (+ 1886us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:01.275995 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:01.276008 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:01.276299 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:01.276631 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:01.276635 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:01.276638 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:01.276781 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:01.277210 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:18:01.277216 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:18:01.277454 (+ 238us) client_negotiation.cc:769] Sending connection context
0618 22:18:01.277650 (+ 196us) client_negotiation.cc:240] Negotiation successful
0618 22:18:01.277990 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":148,"threads_started":1}
I20260618 22:18:01.280512 3161 heartbeater.cc:347] Connected to a master server at 127.0.98.126:46699
I20260618 22:18:01.280766 3161 heartbeater.cc:464] Registering TS with master...
I20260618 22:18:01.281239 3161 heartbeater.cc:511] Master 127.0.98.126:46699 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.67@KRBTEST.COM; defaulting to no policy
I20260618 22:18:01.282509 2836 ts_manager.cc:194] Registered new tserver with Master: b752c702652048d39aedd8a1fdc5e475 (127.0.98.66:40183)
I20260618 22:18:01.283102 2836 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.66@KRBTEST.COM'} at 127.0.98.66:48241
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab.
I20260618 22:18:01.301244 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:46699
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:46085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:18:01.415042 3166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:18:01.415294 3166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:18:01.415367 3166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:18:01.419847 3166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:18:01.419914 3166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:18:01.419973 3166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260618 22:18:01.425458 3166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:46085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:46699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:18:01.426599 3166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:18:01.427626 3166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:18:01.437614 3172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:01.437614 3171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:01.437645 3174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:01.437664 3166 server_base.cc:1034] running on GCE node
I20260618 22:18:01.438329 3166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:18:01.438936 3166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:18:01.440116 3166 hybrid_clock.cc:648] HybridClock initialized: now 1781821081440101 us; error 32 us; skew 500 ppm
Jun 18 22:18:01 dist-test-slave-g6r4 krb5kdc[2791](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821081, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:18:01.443238 3166 init.cc:377] Logged in from keytab as kudu/127.0.98.67@KRBTEST.COM (short username kudu)
I20260618 22:18:01.444211 3166 webserver.cc:460] Webserver started at http://127.0.98.67:46087/ using document root <none> and password file <none>
I20260618 22:18:01.444780 3166 fs_manager.cc:359] Metadata directory not provided
I20260618 22:18:01.444872 3166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:18:01.445097 3166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:18:01.447269 3166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-2/data/instance:
uuid: "01a4d1c5c62742ab99c70d6effb98599"
format_stamp: "Formatted at 2026-06-18 22:18:01 on dist-test-slave-g6r4"
server_key: "07a543de69118ed497176f22b333ef64"
server_key_iv: "f4239b320ded425c36369f810c847ffa"
server_key_version: "encryptionkey@0"
I20260618 22:18:01.447824 3166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-2/wal/instance:
uuid: "01a4d1c5c62742ab99c70d6effb98599"
format_stamp: "Formatted at 2026-06-18 22:18:01 on dist-test-slave-g6r4"
server_key: "07a543de69118ed497176f22b333ef64"
server_key_iv: "f4239b320ded425c36369f810c847ffa"
server_key_version: "encryptionkey@0"
I20260618 22:18:01.451613 3166 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.003s
I20260618 22:18:01.454246 3181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:01.455420 3166 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260618 22:18:01.455570 3166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-2/wal
uuid: "01a4d1c5c62742ab99c70d6effb98599"
format_stamp: "Formatted at 2026-06-18 22:18:01 on dist-test-slave-g6r4"
server_key: "07a543de69118ed497176f22b333ef64"
server_key_iv: "f4239b320ded425c36369f810c847ffa"
server_key_version: "encryptionkey@0"
I20260618 22:18:01.455682 3166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:18:01.468345 3166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:18:01.471514 3166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:18:01.471712 3166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:18:01.472257 3166 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:18:01.473043 3166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:18:01.473114 3166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:01.473182 3166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:18:01.473232 3166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:01.481978 3166 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:38433
I20260618 22:18:01.482060 3294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:38433 every 8 connection(s)
I20260618 22:18:01.482895 3166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
I20260618 22:18:01.487679 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 3166
I20260618 22:18:01.487798 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-2/wal/instance
I20260618 22:18:01.488081 393 external_mini_cluster.cc:1428] Setting key 2d8f69f4433ba4febd3d45089919c54e
Jun 18 22:18:01 dist-test-slave-g6r4 krb5kdc[2791](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821081, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:18:01.496371 2875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:01.484628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:54193 (local address 127.0.98.126:46699)
0618 22:18:01.484793 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:01.484797 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:01.485589 (+ 792us) server_negotiation.cc:407] Connection header received
0618 22:18:01.486966 (+ 1377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:01.486969 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:01.487014 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:01.487124 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:01.488607 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:01.489231 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:01.490657 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:01.490871 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:01.490951 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:01.493177 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:01.493180 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:01.493183 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:01.493203 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:01.494632 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:01.495073 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:01.495076 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:01.495077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:01.495173 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:01.495451 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:01.495453 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:01.495454 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:01.495617 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:01.495698 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:01.496161 (+ 463us) server_negotiation.cc:299] Negotiation successful
0618 22:18:01.496250 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260618 22:18:01.497066 3297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:01.485025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:46699 (local address 127.0.98.67:54193)
0618 22:18:01.485436 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:01.485476 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:01.486775 (+ 1299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:01.487265 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:01.487271 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:01.487785 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:01.488426 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:01.488443 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:01.489382 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:01.489385 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:01.490560 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:01.490569 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:01.490988 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:01.490991 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:01.491122 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:01.491624 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:01.491639 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:01.493039 (+ 1400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:18:01.494728 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:01.494732 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:01.494742 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:01.494971 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:01.495257 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:01.495259 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:01.495261 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:01.495375 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:01.495744 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:18:01.495748 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:18:01.495949 (+ 201us) client_negotiation.cc:769] Sending connection context
0618 22:18:01.496093 (+ 144us) client_negotiation.cc:240] Negotiation successful
0618 22:18:01.496309 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":114,"threads_started":1}
I20260618 22:18:01.498205 3295 heartbeater.cc:347] Connected to a master server at 127.0.98.126:46699
I20260618 22:18:01.498437 3295 heartbeater.cc:464] Registering TS with master...
I20260618 22:18:01.498905 3295 heartbeater.cc:511] Master 127.0.98.126:46699 requested a full tablet report, sending...
I20260618 22:18:01.499855 2836 ts_manager.cc:194] Registered new tserver with Master: 01a4d1c5c62742ab99c70d6effb98599 (127.0.98.67:38433)
I20260618 22:18:01.500385 2836 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.67@KRBTEST.COM'} at 127.0.98.67:54193
I20260618 22:18:01.501713 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260618 22:18:01.514385 2875 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:18:01.512920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60886 (local address 127.0.98.126:46699)
0618 22:18:01.513058 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:01.513063 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:01.513276 (+ 213us) server_negotiation.cc:407] Connection header received
0618 22:18:01.513423 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:01.513425 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:01.513467 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:01.513538 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:01.514248 (+ 710us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60886: BlockingRecv error: recv got EOF from 127.0.0.1:60886 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260618 22:18:01.515378 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 2898
I20260618 22:18:01.520663 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 3032
I20260618 22:18:01.526127 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 3166
I20260618 22:18:01.531472 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 2804
2026-06-18T22:18:01Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2119 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781821065380026-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:18:01 dist-test-slave-g6r4 krb5kdc[3307](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:18:01 dist-test-slave-g6r4 krb5kdc[3307](info): set up 2 sockets
Jun 18 22:18:01 dist-test-slave-g6r4 krb5kdc[3307](info): commencing operation
krb5kdc: starting...
W20260618 22:18:02.629761 393 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781821065380026-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781821065380026-393-0/krb5kdc/test-user.keytab.
Jun 18 22:18:02 dist-test-slave-g6r4 krb5kdc[3307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:18:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:18:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260618 22:18:02.745848 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781821065380026-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781821065380026-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781821065380026-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:39139
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:40497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:39139
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:18:02.851624 3320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:18:02.851848 3320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:18:02.851886 3320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:18:02.855921 3320 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:18:02.855969 3320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:18:02.855988 3320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:18:02.856048 3320 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:18:02.856084 3320 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:18:02.861153 3320 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:40497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781821065380026-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781821065380026-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:39139
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:39139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781821065380026-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:18:02.862145 3320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:18:02.863077 3320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260618 22:18:02.871189 3320 server_base.cc:1034] running on GCE node
W20260618 22:18:02.871176 3325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:02.871173 3326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:02.871344 3328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:02.872006 3320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:18:02.872972 3320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:18:02.874159 3320 hybrid_clock.cc:648] HybridClock initialized: now 1781821082874127 us; error 40 us; skew 500 ppm
Jun 18 22:18:02 dist-test-slave-g6r4 krb5kdc[3307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821082, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:18:02.879595 3320 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260618 22:18:02.880471 3320 webserver.cc:460] Webserver started at http://127.0.98.126:40037/ using document root <none> and password file <none>
I20260618 22:18:02.881031 3320 fs_manager.cc:359] Metadata directory not provided
I20260618 22:18:02.881115 3320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:18:02.881349 3320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:18:02.883308 3320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781821065380026-393-0/minicluster-data/master-0/data/instance:
uuid: "97b78f3878c740b0ac60a8c83c8b7973"
format_stamp: "Formatted at 2026-06-18 22:18:02 on dist-test-slave-g6r4"
server_key: "5339cdc155304f0877a87d3e2e7b6e1e"
server_key_iv: "489d40971e5e18ef935ff997aa63d97c"
server_key_version: "encryptionkey@0"
I20260618 22:18:02.883855 3320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781821065380026-393-0/minicluster-data/master-0/wal/instance:
uuid: "97b78f3878c740b0ac60a8c83c8b7973"
format_stamp: "Formatted at 2026-06-18 22:18:02 on dist-test-slave-g6r4"
server_key: "5339cdc155304f0877a87d3e2e7b6e1e"
server_key_iv: "489d40971e5e18ef935ff997aa63d97c"
server_key_version: "encryptionkey@0"
I20260618 22:18:02.887308 3320 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:18:02.890825 3335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:02.892079 3320 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:18:02.892203 3320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781821065380026-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781821065380026-393-0/minicluster-data/master-0/wal
uuid: "97b78f3878c740b0ac60a8c83c8b7973"
format_stamp: "Formatted at 2026-06-18 22:18:02 on dist-test-slave-g6r4"
server_key: "5339cdc155304f0877a87d3e2e7b6e1e"
server_key_iv: "489d40971e5e18ef935ff997aa63d97c"
server_key_version: "encryptionkey@0"
I20260618 22:18:02.892313 3320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781821065380026-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781821065380026-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781821065380026-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:18:02.903535 3320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:18:02.906580 3320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:18:02.906757 3320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:18:02.913228 3320 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:39139
I20260618 22:18:02.913493 3387 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:39139 every 8 connection(s)
I20260618 22:18:02.914225 3320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
I20260618 22:18:02.916678 3388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:18:02.921300 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 3320
I20260618 22:18:02.921396 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781821065380026-393-0/minicluster-data/master-0/wal/instance
I20260618 22:18:02.921659 393 external_mini_cluster.cc:1428] Setting key 7913e7eb7f1a65225d82571404514434
I20260618 22:18:02.922324 3388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97b78f3878c740b0ac60a8c83c8b7973: Bootstrap starting.
I20260618 22:18:02.924605 3388 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 97b78f3878c740b0ac60a8c83c8b7973: Neither blocks nor log segments found. Creating new log.
I20260618 22:18:02.925331 3388 log.cc:824] T 00000000000000000000000000000000 P 97b78f3878c740b0ac60a8c83c8b7973: Log is configured to *not* fsync() on all Append() calls
I20260618 22:18:02.927121 3388 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97b78f3878c740b0ac60a8c83c8b7973: No bootstrap required, opened a new log
Jun 18 22:18:02 dist-test-slave-g6r4 krb5kdc[3307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:18:02.930949 3388 raft_consensus.cc:348] T 00000000000000000000000000000000 P 97b78f3878c740b0ac60a8c83c8b7973 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97b78f3878c740b0ac60a8c83c8b7973" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39139 } }
I20260618 22:18:02.931205 3388 raft_consensus.cc:374] T 00000000000000000000000000000000 P 97b78f3878c740b0ac60a8c83c8b7973 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:18:02.931264 3388 raft_consensus.cc:729] T 00000000000000000000000000000000 P 97b78f3878c740b0ac60a8c83c8b7973 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97b78f3878c740b0ac60a8c83c8b7973, State: Initialized, Role: FOLLOWER
I20260618 22:18:02.931654 3388 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97b78f3878c740b0ac60a8c83c8b7973 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97b78f3878c740b0ac60a8c83c8b7973" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39139 } }
I20260618 22:18:02.931833 3388 raft_consensus.cc:388] T 00000000000000000000000000000000 P 97b78f3878c740b0ac60a8c83c8b7973 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:18:02.931907 3388 raft_consensus.cc:482] T 00000000000000000000000000000000 P 97b78f3878c740b0ac60a8c83c8b7973 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:18:02.932040 3388 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 97b78f3878c740b0ac60a8c83c8b7973 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:18:02.932889 3388 raft_consensus.cc:504] T 00000000000000000000000000000000 P 97b78f3878c740b0ac60a8c83c8b7973 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97b78f3878c740b0ac60a8c83c8b7973" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39139 } }
I20260618 22:18:02.933251 3388 leader_election.cc:302] T 00000000000000000000000000000000 P 97b78f3878c740b0ac60a8c83c8b7973 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 97b78f3878c740b0ac60a8c83c8b7973; no voters:
I20260618 22:18:02.933614 3388 leader_election.cc:288] T 00000000000000000000000000000000 P 97b78f3878c740b0ac60a8c83c8b7973 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:18:02.933732 3393 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 97b78f3878c740b0ac60a8c83c8b7973 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:18:02.933980 3393 raft_consensus.cc:686] T 00000000000000000000000000000000 P 97b78f3878c740b0ac60a8c83c8b7973 [term 1 LEADER]: Becoming Leader. State: Replica: 97b78f3878c740b0ac60a8c83c8b7973, State: Running, Role: LEADER
I20260618 22:18:02.934288 3393 consensus_queue.cc:237] T 00000000000000000000000000000000 P 97b78f3878c740b0ac60a8c83c8b7973 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97b78f3878c740b0ac60a8c83c8b7973" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39139 } }
I20260618 22:18:02.934885 3388 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97b78f3878c740b0ac60a8c83c8b7973 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:18:02.935400 3393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97b78f3878c740b0ac60a8c83c8b7973 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97b78f3878c740b0ac60a8c83c8b7973. Latest consensus state: current_term: 1 leader_uuid: "97b78f3878c740b0ac60a8c83c8b7973" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97b78f3878c740b0ac60a8c83c8b7973" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39139 } } }
I20260618 22:18:02.935541 3393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97b78f3878c740b0ac60a8c83c8b7973 [sys.catalog]: This master's current role is: LEADER
I20260618 22:18:02.935979 3394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97b78f3878c740b0ac60a8c83c8b7973 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "97b78f3878c740b0ac60a8c83c8b7973" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97b78f3878c740b0ac60a8c83c8b7973" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39139 } } }
I20260618 22:18:02.936089 3394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97b78f3878c740b0ac60a8c83c8b7973 [sys.catalog]: This master's current role is: LEADER
I20260618 22:18:02.936424 3391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:02.922920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54688 (local address 127.0.98.126:39139)
0618 22:18:02.923326 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:02.923339 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:02.923371 (+ 32us) server_negotiation.cc:407] Connection header received
0618 22:18:02.923953 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:02.923968 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:02.924274 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:02.924579 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:02.925377 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:02.926561 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:02.927466 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:02.927931 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:02.928095 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:02.930785 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:02.930790 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:02.930803 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:02.930828 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:02.932771 (+ 1943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:02.933272 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:02.933276 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:02.933280 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:02.933428 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:02.933749 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:02.933752 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:02.933753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:02.934246 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:02.934975 (+ 729us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:02.935117 (+ 142us) server_negotiation.cc:299] Negotiation successful
0618 22:18:02.935412 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":120,"threads_started":1}
I20260618 22:18:02.939327 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260618 22:18:02.939445 3408 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 97b78f3878c740b0ac60a8c83c8b7973: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 22:18:02.939543 3408 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 22:18:02.939640 3400 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:18:02.940339 3400 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:18:02.946064 3400 catalog_manager.cc:1347] Generated new cluster ID: b0a72c909f444f9384ea05cadf4493ea
I20260618 22:18:02.946139 3400 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:18:02.967476 3391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:02.961711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54690 (local address 127.0.98.126:39139)
0618 22:18:02.961848 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:02.961853 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:02.962155 (+ 302us) server_negotiation.cc:407] Connection header received
0618 22:18:02.962297 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:02.962300 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:02.962355 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:02.962448 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:02.963029 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:02.963813 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:02.964972 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:02.965322 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:02.965420 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:02.966380 (+ 960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:02.966384 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:02.966387 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:18:02.966978 (+ 591us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:02.967012 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:18:02.967034 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:02.967129 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:02.967254 (+ 125us) server_negotiation.cc:299] Negotiation successful
0618 22:18:02.967316 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260618 22:18:02.976727 3400 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:18:02.977965 3400 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:18:02.990948 3400 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 97b78f3878c740b0ac60a8c83c8b7973: Generated new TSK 0
I20260618 22:18:02.991724 3400 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:18:03.000600 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 3320
2026-06-18T22:18:03Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1467 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:18:03 dist-test-slave-g6r4 krb5kdc[3421](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:18:03 dist-test-slave-g6r4 krb5kdc[3421](info): set up 2 sockets
Jun 18 22:18:03 dist-test-slave-g6r4 krb5kdc[3421](info): commencing operation
krb5kdc: starting...
W20260618 22:18:04.029807 393 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/krb5kdc/test-user.keytab.
Jun 18 22:18:04 dist-test-slave-g6r4 krb5kdc[3421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821084, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:18:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:18:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260618 22:18:04.148003 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:41181
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:41947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:41181
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:18:04.252760 3434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:18:04.253062 3434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:18:04.253142 3434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:18:04.257332 3434 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:18:04.257412 3434 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:18:04.257448 3434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:18:04.257504 3434 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:18:04.257540 3434 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:18:04.262676 3434 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:41947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:41181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:41181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3434
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:18:04.263624 3434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:18:04.264555 3434 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:18:04.272856 3442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:04.272814 3439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:04.272846 3440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:04.272984 3434 server_base.cc:1034] running on GCE node
I20260618 22:18:04.273614 3434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:18:04.274695 3434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:18:04.275888 3434 hybrid_clock.cc:648] HybridClock initialized: now 1781821084275874 us; error 37 us; skew 500 ppm
Jun 18 22:18:04 dist-test-slave-g6r4 krb5kdc[3421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821084, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:18:04.278689 3434 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260618 22:18:04.279515 3434 webserver.cc:460] Webserver started at http://127.0.98.126:40403/ using document root <none> and password file <none>
I20260618 22:18:04.280045 3434 fs_manager.cc:359] Metadata directory not provided
I20260618 22:18:04.280133 3434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:18:04.280365 3434 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:18:04.282426 3434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/minicluster-data/master-0/data/instance:
uuid: "a505a8fc76844268bb0afa80f17933b2"
format_stamp: "Formatted at 2026-06-18 22:18:04 on dist-test-slave-g6r4"
server_key: "570d2ed628b5391c019e26f2e8245645"
server_key_iv: "f92290d5be8d4f84aa2e1e888e8c08aa"
server_key_version: "encryptionkey@0"
I20260618 22:18:04.283007 3434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/minicluster-data/master-0/wal/instance:
uuid: "a505a8fc76844268bb0afa80f17933b2"
format_stamp: "Formatted at 2026-06-18 22:18:04 on dist-test-slave-g6r4"
server_key: "570d2ed628b5391c019e26f2e8245645"
server_key_iv: "f92290d5be8d4f84aa2e1e888e8c08aa"
server_key_version: "encryptionkey@0"
I20260618 22:18:04.286615 3434 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:18:04.289075 3449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:04.290407 3434 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:18:04.290555 3434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/minicluster-data/master-0/wal
uuid: "a505a8fc76844268bb0afa80f17933b2"
format_stamp: "Formatted at 2026-06-18 22:18:04 on dist-test-slave-g6r4"
server_key: "570d2ed628b5391c019e26f2e8245645"
server_key_iv: "f92290d5be8d4f84aa2e1e888e8c08aa"
server_key_version: "encryptionkey@0"
I20260618 22:18:04.290684 3434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:18:04.304389 3434 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:18:04.307201 3434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:18:04.307397 3434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:18:04.314180 3434 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:41181
I20260618 22:18:04.314432 3501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:41181 every 8 connection(s)
I20260618 22:18:04.315124 3434 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
I20260618 22:18:04.317715 3502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:18:04.322817 3502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a505a8fc76844268bb0afa80f17933b2: Bootstrap starting.
I20260618 22:18:04.323202 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 3434
I20260618 22:18:04.323318 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/minicluster-data/master-0/wal/instance
I20260618 22:18:04.323585 393 external_mini_cluster.cc:1428] Setting key 7d2704fc029f13362bb40cd8c20e7c6f
I20260618 22:18:04.325441 3502 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a505a8fc76844268bb0afa80f17933b2: Neither blocks nor log segments found. Creating new log.
I20260618 22:18:04.326210 3502 log.cc:824] T 00000000000000000000000000000000 P a505a8fc76844268bb0afa80f17933b2: Log is configured to *not* fsync() on all Append() calls
I20260618 22:18:04.328217 3502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a505a8fc76844268bb0afa80f17933b2: No bootstrap required, opened a new log
Jun 18 22:18:04 dist-test-slave-g6r4 krb5kdc[3421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821084, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:18:04.332686 3502 raft_consensus.cc:348] T 00000000000000000000000000000000 P a505a8fc76844268bb0afa80f17933b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a505a8fc76844268bb0afa80f17933b2" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41181 } }
I20260618 22:18:04.332975 3502 raft_consensus.cc:374] T 00000000000000000000000000000000 P a505a8fc76844268bb0afa80f17933b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:18:04.333062 3502 raft_consensus.cc:729] T 00000000000000000000000000000000 P a505a8fc76844268bb0afa80f17933b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a505a8fc76844268bb0afa80f17933b2, State: Initialized, Role: FOLLOWER
I20260618 22:18:04.333595 3502 consensus_queue.cc:260] T 00000000000000000000000000000000 P a505a8fc76844268bb0afa80f17933b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a505a8fc76844268bb0afa80f17933b2" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41181 } }
I20260618 22:18:04.333755 3502 raft_consensus.cc:388] T 00000000000000000000000000000000 P a505a8fc76844268bb0afa80f17933b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:18:04.333820 3502 raft_consensus.cc:482] T 00000000000000000000000000000000 P a505a8fc76844268bb0afa80f17933b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:18:04.333941 3502 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a505a8fc76844268bb0afa80f17933b2 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:18:04.335160 3502 raft_consensus.cc:504] T 00000000000000000000000000000000 P a505a8fc76844268bb0afa80f17933b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a505a8fc76844268bb0afa80f17933b2" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41181 } }
I20260618 22:18:04.335538 3502 leader_election.cc:302] T 00000000000000000000000000000000 P a505a8fc76844268bb0afa80f17933b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a505a8fc76844268bb0afa80f17933b2; no voters:
I20260618 22:18:04.335935 3502 leader_election.cc:288] T 00000000000000000000000000000000 P a505a8fc76844268bb0afa80f17933b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:18:04.336035 3507 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a505a8fc76844268bb0afa80f17933b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:18:04.336316 3507 raft_consensus.cc:686] T 00000000000000000000000000000000 P a505a8fc76844268bb0afa80f17933b2 [term 1 LEADER]: Becoming Leader. State: Replica: a505a8fc76844268bb0afa80f17933b2, State: Running, Role: LEADER
I20260618 22:18:04.336655 3507 consensus_queue.cc:237] T 00000000000000000000000000000000 P a505a8fc76844268bb0afa80f17933b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a505a8fc76844268bb0afa80f17933b2" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41181 } }
I20260618 22:18:04.337261 3502 sys_catalog.cc:564] T 00000000000000000000000000000000 P a505a8fc76844268bb0afa80f17933b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:18:04.338377 3509 sys_catalog.cc:455] T 00000000000000000000000000000000 P a505a8fc76844268bb0afa80f17933b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a505a8fc76844268bb0afa80f17933b2. Latest consensus state: current_term: 1 leader_uuid: "a505a8fc76844268bb0afa80f17933b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a505a8fc76844268bb0afa80f17933b2" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41181 } } }
I20260618 22:18:04.338377 3508 sys_catalog.cc:455] T 00000000000000000000000000000000 P a505a8fc76844268bb0afa80f17933b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a505a8fc76844268bb0afa80f17933b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a505a8fc76844268bb0afa80f17933b2" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41181 } } }
I20260618 22:18:04.338522 3508 sys_catalog.cc:458] T 00000000000000000000000000000000 P a505a8fc76844268bb0afa80f17933b2 [sys.catalog]: This master's current role is: LEADER
I20260618 22:18:04.338522 3509 sys_catalog.cc:458] T 00000000000000000000000000000000 P a505a8fc76844268bb0afa80f17933b2 [sys.catalog]: This master's current role is: LEADER
I20260618 22:18:04.338958 3505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:04.324876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32882 (local address 127.0.98.126:41181)
0618 22:18:04.325275 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:04.325290 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:04.325323 (+ 33us) server_negotiation.cc:407] Connection header received
0618 22:18:04.326041 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:04.326067 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:04.326412 (+ 345us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:04.326750 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:04.327584 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:04.328614 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:04.329794 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:04.330328 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:04.330468 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:04.333325 (+ 2857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:04.333331 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:04.333346 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:04.333377 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:04.335675 (+ 2298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:04.336157 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:04.336167 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:04.336177 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:04.336318 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:04.336631 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:04.336634 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:04.336636 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:04.337110 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:04.337295 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:04.337602 (+ 307us) server_negotiation.cc:299] Negotiation successful
0618 22:18:04.337931 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264,"thread_start_us":129,"threads_started":1}
I20260618 22:18:04.339003 3516 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:18:04.342006 3516 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:18:04.346544 3516 catalog_manager.cc:1347] Generated new cluster ID: 59062279e6344fbdba15f49563814bfc
I20260618 22:18:04.346603 3516 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:18:04.359493 3516 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:18:04.360388 3516 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:18:04.368326 3516 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a505a8fc76844268bb0afa80f17933b2: Generated new TSK 0
I20260618 22:18:04.369086 3516 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab.
I20260618 22:18:04.404743 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:41181
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:41947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:18:04.510766 3528 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:18:04.511023 3528 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:18:04.511101 3528 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:18:04.515009 3528 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:18:04.515079 3528 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:18:04.515162 3528 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260618 22:18:04.520166 3528 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:41947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:41181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3528
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:18:04.521095 3528 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:18:04.522099 3528 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:18:04.530699 3533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:04.530730 3536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:04.530741 3534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:04.531008 3528 server_base.cc:1034] running on GCE node
I20260618 22:18:04.531332 3528 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:18:04.531889 3528 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:18:04.533042 3528 hybrid_clock.cc:648] HybridClock initialized: now 1781821084533017 us; error 36 us; skew 500 ppm
Jun 18 22:18:04 dist-test-slave-g6r4 krb5kdc[3421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821084, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:18:04.535629 3528 init.cc:377] Logged in from keytab as kudu/127.0.98.65@KRBTEST.COM (short username kudu)
I20260618 22:18:04.536376 3528 webserver.cc:460] Webserver started at http://127.0.98.65:39257/ using document root <none> and password file <none>
I20260618 22:18:04.536911 3528 fs_manager.cc:359] Metadata directory not provided
I20260618 22:18:04.536978 3528 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:18:04.537177 3528 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:18:04.539217 3528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-0/data/instance:
uuid: "c6278dfc88444cdf960a52457a629272"
format_stamp: "Formatted at 2026-06-18 22:18:04 on dist-test-slave-g6r4"
server_key: "3cc955205c5b9fe2798be2f6f16f0524"
server_key_iv: "e8cd9d6af0ec9d5bc2f49e2eef131188"
server_key_version: "encryptionkey@0"
I20260618 22:18:04.539702 3528 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-0/wal/instance:
uuid: "c6278dfc88444cdf960a52457a629272"
format_stamp: "Formatted at 2026-06-18 22:18:04 on dist-test-slave-g6r4"
server_key: "3cc955205c5b9fe2798be2f6f16f0524"
server_key_iv: "e8cd9d6af0ec9d5bc2f49e2eef131188"
server_key_version: "encryptionkey@0"
I20260618 22:18:04.543032 3528 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:18:04.545199 3543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:04.546362 3528 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:18:04.546485 3528 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-0/wal
uuid: "c6278dfc88444cdf960a52457a629272"
format_stamp: "Formatted at 2026-06-18 22:18:04 on dist-test-slave-g6r4"
server_key: "3cc955205c5b9fe2798be2f6f16f0524"
server_key_iv: "e8cd9d6af0ec9d5bc2f49e2eef131188"
server_key_version: "encryptionkey@0"
I20260618 22:18:04.546586 3528 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:18:04.570983 3528 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:18:04.574230 3528 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:18:04.574426 3528 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:18:04.574978 3528 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:18:04.575748 3528 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:18:04.575815 3528 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:04.575882 3528 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:18:04.575934 3528 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:04.584367 3528 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:39749
I20260618 22:18:04.584410 3656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:39749 every 8 connection(s)
I20260618 22:18:04.585239 3528 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
I20260618 22:18:04.590723 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 3528
I20260618 22:18:04.590847 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-0/wal/instance
I20260618 22:18:04.591107 393 external_mini_cluster.cc:1428] Setting key 16e37f0a7671b5c853a1c8dcdb452f0e
Jun 18 22:18:04 dist-test-slave-g6r4 krb5kdc[3421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821084, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:18:04.599395 3505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:04.586948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:53183 (local address 127.0.98.126:41181)
0618 22:18:04.587081 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:04.587084 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:04.587865 (+ 781us) server_negotiation.cc:407] Connection header received
0618 22:18:04.589272 (+ 1407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:04.589275 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:04.589333 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:04.589411 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:04.590545 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:04.591196 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:04.592714 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:04.592955 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:04.593065 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:04.595589 (+ 2524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:04.595596 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:04.595602 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:04.595635 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:04.597309 (+ 1674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:04.597783 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:04.597788 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:04.597791 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:04.597886 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:04.598235 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:04.598240 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:04.598243 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:04.598468 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:04.598597 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:04.599066 (+ 469us) server_negotiation.cc:299] Negotiation successful
0618 22:18:04.599197 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260618 22:18:04.600099 3659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:04.587253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:41181 (local address 127.0.98.65:53183)
0618 22:18:04.587697 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:04.587744 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:04.589036 (+ 1292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:04.589567 (+ 531us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:04.589574 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:04.589925 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:04.590382 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:04.590394 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:04.591326 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:04.591335 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:04.592597 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:04.592606 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:04.593105 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:04.593109 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:04.593282 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:04.593954 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:04.593979 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:04.595424 (+ 1445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:18:04.597433 (+ 2009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:04.597437 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:04.597447 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:04.597683 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:04.598005 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:04.598007 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:04.598008 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:04.598125 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:04.598606 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:18:04.598611 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:18:04.598822 (+ 211us) client_negotiation.cc:769] Sending connection context
0618 22:18:04.598990 (+ 168us) client_negotiation.cc:240] Negotiation successful
0618 22:18:04.599240 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":116,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.66@KRBTEST.COM; defaulting to no policy
I20260618 22:18:04.601356 3657 heartbeater.cc:347] Connected to a master server at 127.0.98.126:41181
I20260618 22:18:04.601625 3657 heartbeater.cc:464] Registering TS with master...
I20260618 22:18:04.602201 3657 heartbeater.cc:511] Master 127.0.98.126:41181 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.66@KRBTEST.COM" created.
I20260618 22:18:04.603682 3466 ts_manager.cc:194] Registered new tserver with Master: c6278dfc88444cdf960a52457a629272 (127.0.98.65:39749)
I20260618 22:18:04.604943 3466 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.65@KRBTEST.COM'} at 127.0.98.65:53183
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab.
I20260618 22:18:04.619227 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:41181
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:41947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:18:04.727167 3662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:18:04.727474 3662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:18:04.727531 3662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:18:04.734016 3662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:18:04.734093 3662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:18:04.734215 3662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260618 22:18:04.740216 3662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:41947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:41181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3662
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:18:04.741142 3662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:18:04.742367 3662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:18:04.753374 3670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:04.753588 3662 server_base.cc:1034] running on GCE node
W20260618 22:18:04.753505 3667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:04.753461 3668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:04.754179 3662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:18:04.754768 3662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:18:04.755932 3662 hybrid_clock.cc:648] HybridClock initialized: now 1781821084755901 us; error 42 us; skew 500 ppm
Jun 18 22:18:04 dist-test-slave-g6r4 krb5kdc[3421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821084, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:18:04.758581 3662 init.cc:377] Logged in from keytab as kudu/127.0.98.66@KRBTEST.COM (short username kudu)
I20260618 22:18:04.759423 3662 webserver.cc:460] Webserver started at http://127.0.98.66:41891/ using document root <none> and password file <none>
I20260618 22:18:04.759945 3662 fs_manager.cc:359] Metadata directory not provided
I20260618 22:18:04.760004 3662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:18:04.760247 3662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:18:04.762259 3662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-1/data/instance:
uuid: "586544c1765a4274b8122676f20cab43"
format_stamp: "Formatted at 2026-06-18 22:18:04 on dist-test-slave-g6r4"
server_key: "76235316206fd78842a02b0d309a34bb"
server_key_iv: "22a5d3acf9420b57e115abf025400f49"
server_key_version: "encryptionkey@0"
I20260618 22:18:04.762850 3662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-1/wal/instance:
uuid: "586544c1765a4274b8122676f20cab43"
format_stamp: "Formatted at 2026-06-18 22:18:04 on dist-test-slave-g6r4"
server_key: "76235316206fd78842a02b0d309a34bb"
server_key_iv: "22a5d3acf9420b57e115abf025400f49"
server_key_version: "encryptionkey@0"
I20260618 22:18:04.766371 3662 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260618 22:18:04.768640 3677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:04.769833 3662 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:18:04.769989 3662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-1/wal
uuid: "586544c1765a4274b8122676f20cab43"
format_stamp: "Formatted at 2026-06-18 22:18:04 on dist-test-slave-g6r4"
server_key: "76235316206fd78842a02b0d309a34bb"
server_key_iv: "22a5d3acf9420b57e115abf025400f49"
server_key_version: "encryptionkey@0"
I20260618 22:18:04.770098 3662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:18:04.784459 3662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:18:04.787262 3662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:18:04.787448 3662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:18:04.787985 3662 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:18:04.788825 3662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:18:04.788890 3662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:04.788959 3662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:18:04.789009 3662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:04.797981 3662 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:33057
I20260618 22:18:04.798033 3790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:33057 every 8 connection(s)
I20260618 22:18:04.798856 3662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
I20260618 22:18:04.805243 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 3662
I20260618 22:18:04.805399 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-1/wal/instance
I20260618 22:18:04.805675 393 external_mini_cluster.cc:1428] Setting key 5c09793c0a45fda2688a01271ab01e91
Jun 18 22:18:04 dist-test-slave-g6r4 krb5kdc[3421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821084, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:18:04.813220 3505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:04.800622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:36553 (local address 127.0.98.126:41181)
0618 22:18:04.800777 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:04.800781 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:04.801507 (+ 726us) server_negotiation.cc:407] Connection header received
0618 22:18:04.802952 (+ 1445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:04.802956 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:04.803009 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:04.803123 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:04.804183 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:04.804707 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:04.805968 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:04.806289 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:04.806400 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:04.809337 (+ 2937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:04.809341 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:04.809344 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:04.809368 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:04.810922 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:04.811568 (+ 646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:04.811571 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:04.811572 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:04.811663 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:04.812089 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:04.812092 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:04.812093 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:04.812277 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:04.812373 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:04.812993 (+ 620us) server_negotiation.cc:299] Negotiation successful
0618 22:18:04.813108 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260618 22:18:04.814352 3793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:04.800937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:41181 (local address 127.0.98.66:36553)
0618 22:18:04.801363 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:04.801402 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:04.802756 (+ 1354us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:04.803273 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:04.803281 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:04.803620 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:04.804047 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:04.804059 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:04.804829 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:04.804832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:04.805817 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:04.805826 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:04.806429 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:04.806433 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:04.806597 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:04.807186 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:04.807206 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:04.809149 (+ 1943us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:18:04.811040 (+ 1891us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:04.811049 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:04.811066 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:04.811451 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:04.811760 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:04.811766 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:04.811771 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:04.811954 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:04.812388 (+ 434us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:18:04.812397 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:18:04.812718 (+ 321us) client_negotiation.cc:769] Sending connection context
0618 22:18:04.812948 (+ 230us) client_negotiation.cc:240] Negotiation successful
0618 22:18:04.813299 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":114,"threads_started":1}
I20260618 22:18:04.815503 3791 heartbeater.cc:347] Connected to a master server at 127.0.98.126:41181
WARNING: no policy specified for kudu/127.0.98.67@KRBTEST.COM; defaulting to no policy
I20260618 22:18:04.815768 3791 heartbeater.cc:464] Registering TS with master...
I20260618 22:18:04.816231 3791 heartbeater.cc:511] Master 127.0.98.126:41181 requested a full tablet report, sending...
I20260618 22:18:04.817210 3466 ts_manager.cc:194] Registered new tserver with Master: 586544c1765a4274b8122676f20cab43 (127.0.98.66:33057)
I20260618 22:18:04.817838 3466 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.66@KRBTEST.COM'} at 127.0.98.66:36553
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab.
I20260618 22:18:04.832892 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:41181
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:41947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:18:04.947383 3796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:18:04.947623 3796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:18:04.947662 3796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:18:04.951694 3796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:18:04.951742 3796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:18:04.951797 3796 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260618 22:18:04.956830 3796 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:41947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:41181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3796
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:18:04.957755 3796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:18:04.958729 3796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:18:04.968276 3801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:04.968276 3804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:04.968525 3796 server_base.cc:1034] running on GCE node
W20260618 22:18:04.968276 3802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:04.968900 3796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:18:04.969487 3796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:18:04.970635 3796 hybrid_clock.cc:648] HybridClock initialized: now 1781821084970623 us; error 29 us; skew 500 ppm
Jun 18 22:18:04 dist-test-slave-g6r4 krb5kdc[3421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821084, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:18:04.973196 3796 init.cc:377] Logged in from keytab as kudu/127.0.98.67@KRBTEST.COM (short username kudu)
I20260618 22:18:04.974076 3796 webserver.cc:460] Webserver started at http://127.0.98.67:46657/ using document root <none> and password file <none>
I20260618 22:18:04.974560 3796 fs_manager.cc:359] Metadata directory not provided
I20260618 22:18:04.974615 3796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:18:04.974762 3796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:18:04.976580 3796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-2/data/instance:
uuid: "c05958c8fca0418085e735b17be4b4cc"
format_stamp: "Formatted at 2026-06-18 22:18:04 on dist-test-slave-g6r4"
server_key: "e671f46fd11019341edd3f46ba0d837f"
server_key_iv: "1395806317888522ad4398787ed725e8"
server_key_version: "encryptionkey@0"
I20260618 22:18:04.977059 3796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-2/wal/instance:
uuid: "c05958c8fca0418085e735b17be4b4cc"
format_stamp: "Formatted at 2026-06-18 22:18:04 on dist-test-slave-g6r4"
server_key: "e671f46fd11019341edd3f46ba0d837f"
server_key_iv: "1395806317888522ad4398787ed725e8"
server_key_version: "encryptionkey@0"
I20260618 22:18:04.980758 3796 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260618 22:18:04.983006 3811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:04.984169 3796 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260618 22:18:04.984302 3796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-2/wal
uuid: "c05958c8fca0418085e735b17be4b4cc"
format_stamp: "Formatted at 2026-06-18 22:18:04 on dist-test-slave-g6r4"
server_key: "e671f46fd11019341edd3f46ba0d837f"
server_key_iv: "1395806317888522ad4398787ed725e8"
server_key_version: "encryptionkey@0"
I20260618 22:18:04.984403 3796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:18:04.996342 3796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:18:04.999048 3796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:18:04.999230 3796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:18:04.999728 3796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:18:05.000465 3796 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:18:05.000533 3796 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:05.000593 3796 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:18:05.000641 3796 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:05.009514 3796 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:45511
I20260618 22:18:05.009541 3924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:45511 every 8 connection(s)
I20260618 22:18:05.010385 3796 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
I20260618 22:18:05.018563 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 3796
I20260618 22:18:05.018654 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781821065380026-393-0/minicluster-data/ts-2/wal/instance
I20260618 22:18:05.018956 393 external_mini_cluster.cc:1428] Setting key cc5bde45fb3a331e34f7156c9027a955
Jun 18 22:18:05 dist-test-slave-g6r4 krb5kdc[3421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821084, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:18:05.023294 3505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:05.011998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:42005 (local address 127.0.98.126:41181)
0618 22:18:05.012180 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:05.012185 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:05.012914 (+ 729us) server_negotiation.cc:407] Connection header received
0618 22:18:05.014337 (+ 1423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:05.014340 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:05.014386 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:05.014459 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:05.015515 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:05.016023 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:05.017067 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:05.017267 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:05.017347 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:05.020036 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:05.020042 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:05.020045 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:05.020073 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:05.021646 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:05.022146 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:05.022148 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:05.022149 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:05.022218 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:05.022467 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:05.022468 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:05.022469 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:05.022611 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:05.022679 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:05.023084 (+ 405us) server_negotiation.cc:299] Negotiation successful
0618 22:18:05.023175 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260618 22:18:05.023974 3927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:05.012322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:41181 (local address 127.0.98.67:42005)
0618 22:18:05.012762 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:05.012800 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:05.014154 (+ 1354us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:05.014605 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:05.014648 (+ 43us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:05.014976 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:05.015400 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:05.015411 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:05.016135 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:05.016137 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:05.016969 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:05.016977 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:05.017390 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:05.017393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:05.017523 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:05.018188 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:05.018204 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:05.019889 (+ 1685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:18:05.021742 (+ 1853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:05.021745 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:05.021755 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:05.022064 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:05.022291 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:05.022293 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:05.022294 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:05.022410 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:05.022689 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:18:05.022693 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:18:05.022886 (+ 193us) client_negotiation.cc:769] Sending connection context
0618 22:18:05.023021 (+ 135us) client_negotiation.cc:240] Negotiation successful
0618 22:18:05.023253 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":108,"threads_started":1}
I20260618 22:18:05.025048 3925 heartbeater.cc:347] Connected to a master server at 127.0.98.126:41181
I20260618 22:18:05.025241 3925 heartbeater.cc:464] Registering TS with master...
I20260618 22:18:05.025627 3925 heartbeater.cc:511] Master 127.0.98.126:41181 requested a full tablet report, sending...
I20260618 22:18:05.026599 3466 ts_manager.cc:194] Registered new tserver with Master: c05958c8fca0418085e735b17be4b4cc (127.0.98.67:45511)
I20260618 22:18:05.027168 3466 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.67@KRBTEST.COM'} at 127.0.98.67:42005
I20260618 22:18:05.032437 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 18 22:18:05 dist-test-slave-g6r4 krb5kdc[3421](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821085, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 18 22:18:05 dist-test-slave-g6r4 krb5kdc[3421](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821085, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:18:05.056675 3505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:05.048404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32898 (local address 127.0.98.126:41181)
0618 22:18:05.048545 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:05.048550 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:05.048769 (+ 219us) server_negotiation.cc:407] Connection header received
0618 22:18:05.048925 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:05.048929 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:05.048972 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:05.049060 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:05.049741 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:05.050343 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:05.051229 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:05.051463 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:05.051578 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:05.053794 (+ 2216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:05.053797 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:05.053800 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:05.053830 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:05.055079 (+ 1249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:05.055477 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:05.055481 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:05.055483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:05.055562 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:05.055829 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:05.055832 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:05.055834 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:05.056033 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:05.056184 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:05.056387 (+ 203us) server_negotiation.cc:299] Negotiation successful
0618 22:18:05.056490 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260618 22:18:05.057108 3466 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:32898
I20260618 22:18:05.057996 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 3528
I20260618 22:18:05.063112 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 3662
I20260618 22:18:05.068378 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 3796
I20260618 22:18:05.073443 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 3434
2026-06-18T22:18:05Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2074 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:18:05 dist-test-slave-g6r4 krb5kdc[3937](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:18:05 dist-test-slave-g6r4 krb5kdc[3937](info): set up 2 sockets
Jun 18 22:18:05 dist-test-slave-g6r4 krb5kdc[3937](info): commencing operation
krb5kdc: starting...
W20260618 22:18:06.110419 393 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/krb5kdc/test-user.keytab.
Jun 18 22:18:06 dist-test-slave-g6r4 krb5kdc[3937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:18:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:18:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260618 22:18:06.226799 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:34189
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:36547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:34189
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:18:06.330698 3950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:18:06.330989 3950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:18:06.331029 3950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:18:06.334854 3950 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:18:06.334893 3950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:18:06.334913 3950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:18:06.334928 3950 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:18:06.334944 3950 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:18:06.339808 3950 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:36547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:34189
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:34189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3950
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:18:06.340648 3950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:18:06.341574 3950 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:18:06.349468 3956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:06.350092 3955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:06.350116 3958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:06.350198 3950 server_base.cc:1034] running on GCE node
I20260618 22:18:06.350696 3950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:18:06.351660 3950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:18:06.352814 3950 hybrid_clock.cc:648] HybridClock initialized: now 1781821086352797 us; error 32 us; skew 500 ppm
Jun 18 22:18:06 dist-test-slave-g6r4 krb5kdc[3937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821086, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:18:06.355435 3950 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260618 22:18:06.356216 3950 webserver.cc:460] Webserver started at http://127.0.98.126:45545/ using document root <none> and password file <none>
I20260618 22:18:06.356724 3950 fs_manager.cc:359] Metadata directory not provided
I20260618 22:18:06.356782 3950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:18:06.356977 3950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:18:06.358932 3950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/minicluster-data/master-0/data/instance:
uuid: "66bd01b4e3774110bb24ff9015b64dc3"
format_stamp: "Formatted at 2026-06-18 22:18:06 on dist-test-slave-g6r4"
server_key: "8ff354efc2e35d84cfd653565c999287"
server_key_iv: "5b1f6387c13ce46cc3b2d4415b1a6fe6"
server_key_version: "encryptionkey@0"
I20260618 22:18:06.359467 3950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/minicluster-data/master-0/wal/instance:
uuid: "66bd01b4e3774110bb24ff9015b64dc3"
format_stamp: "Formatted at 2026-06-18 22:18:06 on dist-test-slave-g6r4"
server_key: "8ff354efc2e35d84cfd653565c999287"
server_key_iv: "5b1f6387c13ce46cc3b2d4415b1a6fe6"
server_key_version: "encryptionkey@0"
I20260618 22:18:06.362951 3950 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260618 22:18:06.365097 3965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:06.366321 3950 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:18:06.366447 3950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/minicluster-data/master-0/wal
uuid: "66bd01b4e3774110bb24ff9015b64dc3"
format_stamp: "Formatted at 2026-06-18 22:18:06 on dist-test-slave-g6r4"
server_key: "8ff354efc2e35d84cfd653565c999287"
server_key_iv: "5b1f6387c13ce46cc3b2d4415b1a6fe6"
server_key_version: "encryptionkey@0"
I20260618 22:18:06.366550 3950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:18:06.384562 3950 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:18:06.387223 3950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:18:06.387423 3950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:18:06.394032 3950 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:34189
I20260618 22:18:06.394047 4017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:34189 every 8 connection(s)
I20260618 22:18:06.394927 3950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
I20260618 22:18:06.397507 4018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:18:06.402086 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 3950
I20260618 22:18:06.402218 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/minicluster-data/master-0/wal/instance
I20260618 22:18:06.402457 393 external_mini_cluster.cc:1428] Setting key a5d97ec5e8c977aee5fc797c76b3b8ad
I20260618 22:18:06.402745 4018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66bd01b4e3774110bb24ff9015b64dc3: Bootstrap starting.
I20260618 22:18:06.405134 4018 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66bd01b4e3774110bb24ff9015b64dc3: Neither blocks nor log segments found. Creating new log.
I20260618 22:18:06.406116 4018 log.cc:824] T 00000000000000000000000000000000 P 66bd01b4e3774110bb24ff9015b64dc3: Log is configured to *not* fsync() on all Append() calls
I20260618 22:18:06.408034 4018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66bd01b4e3774110bb24ff9015b64dc3: No bootstrap required, opened a new log
Jun 18 22:18:06 dist-test-slave-g6r4 krb5kdc[3937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:18:06.412305 4018 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66bd01b4e3774110bb24ff9015b64dc3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66bd01b4e3774110bb24ff9015b64dc3" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34189 } }
I20260618 22:18:06.412559 4018 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66bd01b4e3774110bb24ff9015b64dc3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:18:06.412642 4018 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66bd01b4e3774110bb24ff9015b64dc3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66bd01b4e3774110bb24ff9015b64dc3, State: Initialized, Role: FOLLOWER
I20260618 22:18:06.413079 4018 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66bd01b4e3774110bb24ff9015b64dc3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66bd01b4e3774110bb24ff9015b64dc3" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34189 } }
I20260618 22:18:06.413196 4018 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66bd01b4e3774110bb24ff9015b64dc3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:18:06.413285 4018 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66bd01b4e3774110bb24ff9015b64dc3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:18:06.413404 4018 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66bd01b4e3774110bb24ff9015b64dc3 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:18:06.414283 4018 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66bd01b4e3774110bb24ff9015b64dc3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66bd01b4e3774110bb24ff9015b64dc3" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34189 } }
I20260618 22:18:06.414634 4018 leader_election.cc:302] T 00000000000000000000000000000000 P 66bd01b4e3774110bb24ff9015b64dc3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66bd01b4e3774110bb24ff9015b64dc3; no voters:
I20260618 22:18:06.414949 4018 leader_election.cc:288] T 00000000000000000000000000000000 P 66bd01b4e3774110bb24ff9015b64dc3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:18:06.415041 4023 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66bd01b4e3774110bb24ff9015b64dc3 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:18:06.415257 4023 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66bd01b4e3774110bb24ff9015b64dc3 [term 1 LEADER]: Becoming Leader. State: Replica: 66bd01b4e3774110bb24ff9015b64dc3, State: Running, Role: LEADER
I20260618 22:18:06.415611 4023 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66bd01b4e3774110bb24ff9015b64dc3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66bd01b4e3774110bb24ff9015b64dc3" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34189 } }
I20260618 22:18:06.416174 4018 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66bd01b4e3774110bb24ff9015b64dc3 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:18:06.416833 4025 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66bd01b4e3774110bb24ff9015b64dc3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66bd01b4e3774110bb24ff9015b64dc3. Latest consensus state: current_term: 1 leader_uuid: "66bd01b4e3774110bb24ff9015b64dc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66bd01b4e3774110bb24ff9015b64dc3" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34189 } } }
I20260618 22:18:06.416952 4025 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66bd01b4e3774110bb24ff9015b64dc3 [sys.catalog]: This master's current role is: LEADER
I20260618 22:18:06.416992 4021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:06.403739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41120 (local address 127.0.98.126:34189)
0618 22:18:06.404153 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:06.404169 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:06.404201 (+ 32us) server_negotiation.cc:407] Connection header received
0618 22:18:06.404776 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:06.404791 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:06.405086 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:06.405401 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:06.406315 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:06.407415 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:06.408584 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:06.409020 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:06.409135 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:06.411796 (+ 2661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:06.411807 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:06.411819 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:06.411843 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:06.413722 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:06.414199 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:06.414205 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:06.414210 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:06.414334 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:06.414661 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:06.414664 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:06.414665 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:06.415159 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:06.415347 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:06.415764 (+ 417us) server_negotiation.cc:299] Negotiation successful
0618 22:18:06.416045 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":148,"threads_started":1}
I20260618 22:18:06.417235 4024 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66bd01b4e3774110bb24ff9015b64dc3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66bd01b4e3774110bb24ff9015b64dc3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66bd01b4e3774110bb24ff9015b64dc3" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34189 } } }
I20260618 22:18:06.417347 4024 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66bd01b4e3774110bb24ff9015b64dc3 [sys.catalog]: This master's current role is: LEADER
W20260618 22:18:06.420667 4038 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 66bd01b4e3774110bb24ff9015b64dc3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 22:18:06.420733 4038 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 22:18:06.420810 4032 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:18:06.421514 4032 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:18:06.426939 4032 catalog_manager.cc:1347] Generated new cluster ID: 1b0f7a162c30444cb940a74dd18783fa
I20260618 22:18:06.427004 4032 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:18:06.441932 4032 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:18:06.442994 4032 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:18:06.457195 4032 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 66bd01b4e3774110bb24ff9015b64dc3: Generated new TSK 0
I20260618 22:18:06.458029 4032 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab.
I20260618 22:18:06.492331 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:34189
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:36547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:18:06.599120 4044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:18:06.599344 4044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:18:06.599383 4044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:18:06.603427 4044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:18:06.603482 4044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:18:06.603564 4044 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260618 22:18:06.608568 4044 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:36547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:34189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:18:06.609500 4044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:18:06.610509 4044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:18:06.619689 4052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:06.619686 4049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:06.619745 4050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:06.620012 4044 server_base.cc:1034] running on GCE node
I20260618 22:18:06.620440 4044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:18:06.621090 4044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:18:06.622260 4044 hybrid_clock.cc:648] HybridClock initialized: now 1781821086622226 us; error 46 us; skew 500 ppm
Jun 18 22:18:06 dist-test-slave-g6r4 krb5kdc[3937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821086, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:18:06.624788 4044 init.cc:377] Logged in from keytab as kudu/127.0.98.65@KRBTEST.COM (short username kudu)
I20260618 22:18:06.625576 4044 webserver.cc:460] Webserver started at http://127.0.98.65:43457/ using document root <none> and password file <none>
I20260618 22:18:06.626150 4044 fs_manager.cc:359] Metadata directory not provided
I20260618 22:18:06.626258 4044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:18:06.626468 4044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:18:06.628351 4044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/minicluster-data/ts-0/data/instance:
uuid: "4932a10fc8ab4318b6c00de711351f7f"
format_stamp: "Formatted at 2026-06-18 22:18:06 on dist-test-slave-g6r4"
server_key: "6b069cd9d2b4107d852bd0246c3644b0"
server_key_iv: "96b69aa9203da7666105cf3a1272f208"
server_key_version: "encryptionkey@0"
I20260618 22:18:06.628880 4044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/minicluster-data/ts-0/wal/instance:
uuid: "4932a10fc8ab4318b6c00de711351f7f"
format_stamp: "Formatted at 2026-06-18 22:18:06 on dist-test-slave-g6r4"
server_key: "6b069cd9d2b4107d852bd0246c3644b0"
server_key_iv: "96b69aa9203da7666105cf3a1272f208"
server_key_version: "encryptionkey@0"
I20260618 22:18:06.632262 4044 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260618 22:18:06.634521 4059 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:06.635663 4044 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:18:06.635798 4044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/minicluster-data/ts-0/wal
uuid: "4932a10fc8ab4318b6c00de711351f7f"
format_stamp: "Formatted at 2026-06-18 22:18:06 on dist-test-slave-g6r4"
server_key: "6b069cd9d2b4107d852bd0246c3644b0"
server_key_iv: "96b69aa9203da7666105cf3a1272f208"
server_key_version: "encryptionkey@0"
I20260618 22:18:06.635902 4044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:18:06.656210 4044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:18:06.659665 4044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:18:06.659861 4044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:18:06.660426 4044 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:18:06.661204 4044 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:18:06.661252 4044 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:06.661316 4044 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:18:06.661346 4044 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:06.669943 4044 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:37107
I20260618 22:18:06.669981 4172 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:37107 every 8 connection(s)
I20260618 22:18:06.670861 4044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
I20260618 22:18:06.678292 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 4044
I20260618 22:18:06.678443 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/minicluster-data/ts-0/wal/instance
I20260618 22:18:06.678785 393 external_mini_cluster.cc:1428] Setting key 412cb6f3f89e3a57af01fa0e461c6e9a
Jun 18 22:18:06 dist-test-slave-g6r4 krb5kdc[3937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821086, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:18:06.684424 4021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:06.672621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:53977 (local address 127.0.98.126:34189)
0618 22:18:06.672745 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:06.672749 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:06.673524 (+ 775us) server_negotiation.cc:407] Connection header received
0618 22:18:06.674908 (+ 1384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:06.674910 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:06.674964 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:06.675040 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:06.676109 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:06.676627 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:06.677681 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:06.677955 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:06.678045 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:06.680758 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:06.680762 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:06.680764 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:06.680798 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:06.682467 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:06.682945 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:06.682950 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:06.682952 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:06.683038 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:06.683367 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:06.683372 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:06.683374 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:06.683551 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:06.683668 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:06.684138 (+ 470us) server_negotiation.cc:299] Negotiation successful
0618 22:18:06.684257 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260618 22:18:06.685218 4175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:06.672958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34189 (local address 127.0.98.65:53977)
0618 22:18:06.673379 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:06.673418 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:06.674709 (+ 1291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:06.675193 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:06.675199 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:06.675543 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:06.675979 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:06.675991 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:06.676727 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:06.676730 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:06.677569 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:06.677577 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:06.678085 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:06.678087 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:06.678219 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:06.678816 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:06.678831 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:06.680578 (+ 1747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:18:06.682566 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:06.682572 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:06.682592 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:06.682842 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:06.683134 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:06.683138 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:06.683141 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:06.683274 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:06.683675 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:18:06.683684 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:18:06.683909 (+ 225us) client_negotiation.cc:769] Sending connection context
0618 22:18:06.684091 (+ 182us) client_negotiation.cc:240] Negotiation successful
0618 22:18:06.684388 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":112,"threads_started":1}
I20260618 22:18:06.686435 4173 heartbeater.cc:347] Connected to a master server at 127.0.98.126:34189
I20260618 22:18:06.686690 4173 heartbeater.cc:464] Registering TS with master...
I20260618 22:18:06.687247 4173 heartbeater.cc:511] Master 127.0.98.126:34189 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.66@KRBTEST.COM; defaulting to no policy
I20260618 22:18:06.688812 3982 ts_manager.cc:194] Registered new tserver with Master: 4932a10fc8ab4318b6c00de711351f7f (127.0.98.65:37107)
I20260618 22:18:06.690088 3982 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.65@KRBTEST.COM'} at 127.0.98.65:53977
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab.
I20260618 22:18:06.704878 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:34189
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:36547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:18:06.811157 4178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:18:06.811401 4178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:18:06.811441 4178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:18:06.815420 4178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:18:06.815469 4178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:18:06.815564 4178 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260618 22:18:06.820766 4178 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:36547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:34189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4178
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:18:06.821679 4178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:18:06.822646 4178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:18:06.831818 4184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:06.831804 4186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:06.831864 4178 server_base.cc:1034] running on GCE node
W20260618 22:18:06.831806 4183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:06.832337 4178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:18:06.832906 4178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:18:06.834075 4178 hybrid_clock.cc:648] HybridClock initialized: now 1781821086834048 us; error 39 us; skew 500 ppm
Jun 18 22:18:06 dist-test-slave-g6r4 krb5kdc[3937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821086, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:18:06.836627 4178 init.cc:377] Logged in from keytab as kudu/127.0.98.66@KRBTEST.COM (short username kudu)
I20260618 22:18:06.837443 4178 webserver.cc:460] Webserver started at http://127.0.98.66:42163/ using document root <none> and password file <none>
I20260618 22:18:06.838039 4178 fs_manager.cc:359] Metadata directory not provided
I20260618 22:18:06.838125 4178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:18:06.838346 4178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:18:06.840477 4178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/minicluster-data/ts-1/data/instance:
uuid: "c6cee027af81418c8c34f0c52473193f"
format_stamp: "Formatted at 2026-06-18 22:18:06 on dist-test-slave-g6r4"
server_key: "7a82b114bf380f2ea6c415272d4ca422"
server_key_iv: "f67432e6929558ff4dee0144f93f6095"
server_key_version: "encryptionkey@0"
I20260618 22:18:06.841022 4178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/minicluster-data/ts-1/wal/instance:
uuid: "c6cee027af81418c8c34f0c52473193f"
format_stamp: "Formatted at 2026-06-18 22:18:06 on dist-test-slave-g6r4"
server_key: "7a82b114bf380f2ea6c415272d4ca422"
server_key_iv: "f67432e6929558ff4dee0144f93f6095"
server_key_version: "encryptionkey@0"
I20260618 22:18:06.844449 4178 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260618 22:18:06.846829 4193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:06.848052 4178 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:18:06.848178 4178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/minicluster-data/ts-1/wal
uuid: "c6cee027af81418c8c34f0c52473193f"
format_stamp: "Formatted at 2026-06-18 22:18:06 on dist-test-slave-g6r4"
server_key: "7a82b114bf380f2ea6c415272d4ca422"
server_key_iv: "f67432e6929558ff4dee0144f93f6095"
server_key_version: "encryptionkey@0"
I20260618 22:18:06.848291 4178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:18:06.869302 4178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:18:06.872395 4178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:18:06.872578 4178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:18:06.873118 4178 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:18:06.873875 4178 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:18:06.873993 4178 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:06.874074 4178 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:18:06.874125 4178 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:06.883198 4178 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:35041
I20260618 22:18:06.883229 4306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:35041 every 8 connection(s)
I20260618 22:18:06.884037 4178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
I20260618 22:18:06.890699 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 4178
I20260618 22:18:06.890818 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/minicluster-data/ts-1/wal/instance
I20260618 22:18:06.891119 393 external_mini_cluster.cc:1428] Setting key 50a89b3e951225048cee3f0d07668e08
Jun 18 22:18:06 dist-test-slave-g6r4 krb5kdc[3937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821086, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:18:06.898609 4021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:06.885834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:55345 (local address 127.0.98.126:34189)
0618 22:18:06.886070 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:06.886075 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:06.886745 (+ 670us) server_negotiation.cc:407] Connection header received
0618 22:18:06.888114 (+ 1369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:06.888118 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:06.888172 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:06.888251 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:06.889330 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:06.889861 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:06.891323 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:06.891604 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:06.891742 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:06.894498 (+ 2756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:06.894502 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:06.894504 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:06.894528 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:06.896240 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:06.896705 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:06.896711 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:06.896716 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:06.896833 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:06.897152 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:06.897158 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:06.897161 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:06.897418 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:06.897532 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:06.898234 (+ 702us) server_negotiation.cc:299] Negotiation successful
0618 22:18:06.898377 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151}
I20260618 22:18:06.899628 4309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:06.886171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34189 (local address 127.0.98.66:55345)
0618 22:18:06.886600 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:06.886645 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:06.887907 (+ 1262us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:06.888404 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:06.888411 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:06.888741 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:06.889190 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:06.889202 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:06.890023 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:06.890033 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:06.891209 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:06.891219 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:06.891870 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:06.891876 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:06.892062 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:06.892682 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:06.892697 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:06.894298 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:18:06.896355 (+ 2057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:06.896359 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:06.896370 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:06.896620 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:06.896933 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:06.896935 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:06.896936 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:06.897062 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:06.897537 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:18:06.897545 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:18:06.897937 (+ 392us) client_negotiation.cc:769] Sending connection context
0618 22:18:06.898163 (+ 226us) client_negotiation.cc:240] Negotiation successful
0618 22:18:06.898512 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":106,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.67@KRBTEST.COM; defaulting to no policy
I20260618 22:18:06.901239 4307 heartbeater.cc:347] Connected to a master server at 127.0.98.126:34189
I20260618 22:18:06.901477 4307 heartbeater.cc:464] Registering TS with master...
I20260618 22:18:06.902046 4307 heartbeater.cc:511] Master 127.0.98.126:34189 requested a full tablet report, sending...
I20260618 22:18:06.903260 3982 ts_manager.cc:194] Registered new tserver with Master: c6cee027af81418c8c34f0c52473193f (127.0.98.66:35041)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.67@KRBTEST.COM" created.
I20260618 22:18:06.904042 3982 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.66@KRBTEST.COM'} at 127.0.98.66:55345
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab.
I20260618 22:18:06.919286 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:34189
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:36547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:18:07.024646 4312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:18:07.024873 4312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:18:07.024914 4312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:18:07.028748 4312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:18:07.028800 4312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:18:07.028856 4312 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260618 22:18:07.033666 4312 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:36547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:34189
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:18:07.034580 4312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:18:07.035454 4312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260618 22:18:07.044687 4312 server_base.cc:1034] running on GCE node
W20260618 22:18:07.044720 4317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:07.044720 4318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:07.044978 4320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:07.045331 4312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:18:07.046041 4312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:18:07.047216 4312 hybrid_clock.cc:648] HybridClock initialized: now 1781821087047200 us; error 30 us; skew 500 ppm
Jun 18 22:18:07 dist-test-slave-g6r4 krb5kdc[3937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821087, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:18:07.049803 4312 init.cc:377] Logged in from keytab as kudu/127.0.98.67@KRBTEST.COM (short username kudu)
I20260618 22:18:07.050621 4312 webserver.cc:460] Webserver started at http://127.0.98.67:36115/ using document root <none> and password file <none>
I20260618 22:18:07.051162 4312 fs_manager.cc:359] Metadata directory not provided
I20260618 22:18:07.051252 4312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:18:07.051517 4312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:18:07.053498 4312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/minicluster-data/ts-2/data/instance:
uuid: "a1ad06d8abd24380ab55e9606f6c318b"
format_stamp: "Formatted at 2026-06-18 22:18:07 on dist-test-slave-g6r4"
server_key: "6acd0b8ab7ed5ac63ad8a262f1d14aa5"
server_key_iv: "1c87e1be5d0f665a937da45c091b1874"
server_key_version: "encryptionkey@0"
I20260618 22:18:07.054103 4312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/minicluster-data/ts-2/wal/instance:
uuid: "a1ad06d8abd24380ab55e9606f6c318b"
format_stamp: "Formatted at 2026-06-18 22:18:07 on dist-test-slave-g6r4"
server_key: "6acd0b8ab7ed5ac63ad8a262f1d14aa5"
server_key_iv: "1c87e1be5d0f665a937da45c091b1874"
server_key_version: "encryptionkey@0"
I20260618 22:18:07.057631 4312 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260618 22:18:07.059940 4327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:07.061110 4312 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260618 22:18:07.061239 4312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/minicluster-data/ts-2/wal
uuid: "a1ad06d8abd24380ab55e9606f6c318b"
format_stamp: "Formatted at 2026-06-18 22:18:07 on dist-test-slave-g6r4"
server_key: "6acd0b8ab7ed5ac63ad8a262f1d14aa5"
server_key_iv: "1c87e1be5d0f665a937da45c091b1874"
server_key_version: "encryptionkey@0"
I20260618 22:18:07.061355 4312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:18:07.075763 4312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:18:07.078635 4312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:18:07.078819 4312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:18:07.079346 4312 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:18:07.080377 4312 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:18:07.080438 4312 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:07.080488 4312 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:18:07.080510 4312 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:07.090619 4312 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:38765
I20260618 22:18:07.090641 4440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:38765 every 8 connection(s)
I20260618 22:18:07.091449 4312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
I20260618 22:18:07.094874 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 4312
I20260618 22:18:07.094996 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781821065380026-393-0/minicluster-data/ts-2/wal/instance
I20260618 22:18:07.095278 393 external_mini_cluster.cc:1428] Setting key 40e721a09dc770ec10f28848dbfb608f
Jun 18 22:18:07 dist-test-slave-g6r4 krb5kdc[3937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821087, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:18:07.105870 4021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:07.093127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:47529 (local address 127.0.98.126:34189)
0618 22:18:07.093271 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:07.093276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:07.094084 (+ 808us) server_negotiation.cc:407] Connection header received
0618 22:18:07.096090 (+ 2006us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:07.096093 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:07.096169 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:07.096266 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:07.098295 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:07.099087 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:07.100163 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:07.100385 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:07.100465 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:07.102768 (+ 2303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:07.102771 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:07.102773 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:07.102796 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:07.104083 (+ 1287us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:07.104563 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:07.104566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:07.104567 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:07.104674 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:07.104967 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:07.104969 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:07.104970 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:07.105135 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:07.105210 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:07.105624 (+ 414us) server_negotiation.cc:299] Negotiation successful
0618 22:18:07.105754 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260618 22:18:07.106726 4443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:07.093477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34189 (local address 127.0.98.67:47529)
0618 22:18:07.093939 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:07.093989 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:07.095832 (+ 1843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:07.096711 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:07.096720 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:07.097368 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:07.098063 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:07.098116 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:07.099192 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:07.099195 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:07.100048 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:07.100058 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:07.100510 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:07.100512 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:07.100665 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:07.101165 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:07.101179 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:07.102623 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:18:07.104181 (+ 1558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:07.104185 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:07.104196 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:07.104463 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:07.104754 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:07.104757 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:07.104758 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:07.104884 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:07.105218 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:18:07.105222 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:18:07.105427 (+ 205us) client_negotiation.cc:769] Sending connection context
0618 22:18:07.105567 (+ 140us) client_negotiation.cc:240] Negotiation successful
0618 22:18:07.105807 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":132,"threads_started":1}
I20260618 22:18:07.107906 4441 heartbeater.cc:347] Connected to a master server at 127.0.98.126:34189
I20260618 22:18:07.108136 4441 heartbeater.cc:464] Registering TS with master...
I20260618 22:18:07.108685 4441 heartbeater.cc:511] Master 127.0.98.126:34189 requested a full tablet report, sending...
I20260618 22:18:07.109586 3982 ts_manager.cc:194] Registered new tserver with Master: a1ad06d8abd24380ab55e9606f6c318b (127.0.98.67:38765)
I20260618 22:18:07.110163 3982 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.67@KRBTEST.COM'} at 127.0.98.67:47529
I20260618 22:18:07.120375 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 18 22:18:07 dist-test-slave-g6r4 krb5kdc[3937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821087, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 18 22:18:07 dist-test-slave-g6r4 krb5kdc[3937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821087, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.65@KRBTEST.COM
I20260618 22:18:07.144543 4448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:07.135697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50576 (local address 127.0.98.65:37107)
0618 22:18:07.135957 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:07.135971 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:07.135994 (+ 23us) server_negotiation.cc:407] Connection header received
0618 22:18:07.136041 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:07.136046 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:07.136222 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:07.136340 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:07.136938 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:07.137684 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:07.138555 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:07.138927 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:07.139063 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:07.141418 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:07.141426 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:07.141438 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:07.141459 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:07.143020 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:07.143403 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:07.143407 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:07.143409 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:07.143485 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:07.143743 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:07.143745 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:07.143747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:07.143965 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:07.144129 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:07.144279 (+ 150us) server_negotiation.cc:299] Negotiation successful
0618 22:18:07.144399 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":59,"threads_started":1}
Jun 18 22:18:07 dist-test-slave-g6r4 krb5kdc[3937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821087, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:18:07.154811 4021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:07.147291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41136 (local address 127.0.98.126:34189)
0618 22:18:07.147425 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:07.147430 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:07.147513 (+ 83us) server_negotiation.cc:407] Connection header received
0618 22:18:07.147604 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:07.147606 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:07.147648 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:07.147711 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:07.148241 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:07.148749 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:07.149436 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:07.149632 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:07.149750 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:07.151943 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:07.151945 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:07.151947 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:07.151968 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:07.153254 (+ 1286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:07.153605 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:07.153608 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:07.153609 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:07.153715 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:07.154019 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:07.154021 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:07.154023 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:07.154210 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:07.154309 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:07.154503 (+ 194us) server_negotiation.cc:299] Negotiation successful
0618 22:18:07.154589 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
W20260618 22:18:07.155201 3982 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41136
I20260618 22:18:07.155742 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 4044
I20260618 22:18:07.161186 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 4178
I20260618 22:18:07.166567 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 4312
I20260618 22:18:07.171550 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 3950
2026-06-18T22:18:07Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2097 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:18:07 dist-test-slave-g6r4 krb5kdc[4453](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:18:07 dist-test-slave-g6r4 krb5kdc[4453](info): set up 2 sockets
Jun 18 22:18:07 dist-test-slave-g6r4 krb5kdc[4453](info): commencing operation
krb5kdc: starting...
W20260618 22:18:08.214017 393 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/krb5kdc/test-user.keytab.
Jun 18 22:18:08 dist-test-slave-g6r4 krb5kdc[4453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821088, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:18:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:18:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260618 22:18:08.332906 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:38129
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:42987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:38129
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:18:08.439953 4466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:18:08.440209 4466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:18:08.440253 4466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:18:08.444267 4466 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:18:08.444322 4466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:18:08.444341 4466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:18:08.444357 4466 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:18:08.444372 4466 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:18:08.449414 4466 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:42987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:38129
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:38129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:18:08.450423 4466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:18:08.451347 4466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:18:08.459498 4471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:08.459489 4472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:08.459510 4474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:08.459825 4466 server_base.cc:1034] running on GCE node
I20260618 22:18:08.460433 4466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:18:08.461552 4466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:18:08.462738 4466 hybrid_clock.cc:648] HybridClock initialized: now 1781821088462707 us; error 39 us; skew 500 ppm
Jun 18 22:18:08 dist-test-slave-g6r4 krb5kdc[4453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821088, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:18:08.469800 4466 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260618 22:18:08.471024 4466 fs_manager.cc:359] Metadata directory not provided
I20260618 22:18:08.471130 4466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:18:08.471427 4466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:18:08.474881 4466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/minicluster-data/master-0/data/instance:
uuid: "4b84b451dbf94bf5b2e4a90b145c09bb"
format_stamp: "Formatted at 2026-06-18 22:18:08 on dist-test-slave-g6r4"
server_key: "7c453160f95ceb66daa16d3cdff7d8f9"
server_key_iv: "98cebb2bdab216ce1e8ec00fe6bdebd4"
server_key_version: "encryptionkey@0"
I20260618 22:18:08.475629 4466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/minicluster-data/master-0/wal/instance:
uuid: "4b84b451dbf94bf5b2e4a90b145c09bb"
format_stamp: "Formatted at 2026-06-18 22:18:08 on dist-test-slave-g6r4"
server_key: "7c453160f95ceb66daa16d3cdff7d8f9"
server_key_iv: "98cebb2bdab216ce1e8ec00fe6bdebd4"
server_key_version: "encryptionkey@0"
I20260618 22:18:08.481312 4466 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260618 22:18:08.484413 4480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:08.485844 4466 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260618 22:18:08.486012 4466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/minicluster-data/master-0/wal
uuid: "4b84b451dbf94bf5b2e4a90b145c09bb"
format_stamp: "Formatted at 2026-06-18 22:18:08 on dist-test-slave-g6r4"
server_key: "7c453160f95ceb66daa16d3cdff7d8f9"
server_key_iv: "98cebb2bdab216ce1e8ec00fe6bdebd4"
server_key_version: "encryptionkey@0"
I20260618 22:18:08.486138 4466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:18:08.520650 4466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:18:08.523634 4466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:18:08.523840 4466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:18:08.530579 4466 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:38129
I20260618 22:18:08.530844 4532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:38129 every 8 connection(s)
I20260618 22:18:08.531246 4466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
I20260618 22:18:08.533864 4533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:18:08.538749 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 4466
I20260618 22:18:08.538877 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/minicluster-data/master-0/wal/instance
I20260618 22:18:08.539125 393 external_mini_cluster.cc:1428] Setting key 566f1b4ad376c14cf08b4716f5ddf2d3
I20260618 22:18:08.539258 4533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b84b451dbf94bf5b2e4a90b145c09bb: Bootstrap starting.
I20260618 22:18:08.541640 4533 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b84b451dbf94bf5b2e4a90b145c09bb: Neither blocks nor log segments found. Creating new log.
I20260618 22:18:08.542627 4533 log.cc:824] T 00000000000000000000000000000000 P 4b84b451dbf94bf5b2e4a90b145c09bb: Log is configured to *not* fsync() on all Append() calls
I20260618 22:18:08.544647 4533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b84b451dbf94bf5b2e4a90b145c09bb: No bootstrap required, opened a new log
Jun 18 22:18:08 dist-test-slave-g6r4 krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821088, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:18:08.548933 4533 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b84b451dbf94bf5b2e4a90b145c09bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b84b451dbf94bf5b2e4a90b145c09bb" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38129 } }
I20260618 22:18:08.549227 4533 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b84b451dbf94bf5b2e4a90b145c09bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:18:08.549314 4533 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b84b451dbf94bf5b2e4a90b145c09bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b84b451dbf94bf5b2e4a90b145c09bb, State: Initialized, Role: FOLLOWER
I20260618 22:18:08.549783 4533 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b84b451dbf94bf5b2e4a90b145c09bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b84b451dbf94bf5b2e4a90b145c09bb" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38129 } }
I20260618 22:18:08.549954 4533 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b84b451dbf94bf5b2e4a90b145c09bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:18:08.550022 4533 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b84b451dbf94bf5b2e4a90b145c09bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:18:08.550125 4533 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b84b451dbf94bf5b2e4a90b145c09bb [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:18:08.550937 4533 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b84b451dbf94bf5b2e4a90b145c09bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b84b451dbf94bf5b2e4a90b145c09bb" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38129 } }
I20260618 22:18:08.551263 4533 leader_election.cc:302] T 00000000000000000000000000000000 P 4b84b451dbf94bf5b2e4a90b145c09bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b84b451dbf94bf5b2e4a90b145c09bb; no voters:
I20260618 22:18:08.551546 4533 leader_election.cc:288] T 00000000000000000000000000000000 P 4b84b451dbf94bf5b2e4a90b145c09bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:18:08.551635 4538 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b84b451dbf94bf5b2e4a90b145c09bb [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:18:08.551831 4538 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b84b451dbf94bf5b2e4a90b145c09bb [term 1 LEADER]: Becoming Leader. State: Replica: 4b84b451dbf94bf5b2e4a90b145c09bb, State: Running, Role: LEADER
I20260618 22:18:08.552193 4538 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b84b451dbf94bf5b2e4a90b145c09bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b84b451dbf94bf5b2e4a90b145c09bb" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38129 } }
I20260618 22:18:08.552850 4533 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b84b451dbf94bf5b2e4a90b145c09bb [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:18:08.554087 4536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:08.540369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50416 (local address 127.0.98.126:38129)
0618 22:18:08.540730 (+ 361us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:08.540745 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:08.540780 (+ 35us) server_negotiation.cc:407] Connection header received
0618 22:18:08.541386 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:08.541402 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:08.541706 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:08.542093 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:08.542954 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:08.544062 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:08.545264 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:08.545719 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:08.545854 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:08.548510 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:08.548515 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:08.548527 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:08.548552 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:08.550555 (+ 2003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:08.551012 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:08.551019 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:08.551026 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:08.551182 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:08.551474 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:08.551479 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:08.551481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:08.552005 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:08.552195 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:08.552794 (+ 599us) server_negotiation.cc:299] Negotiation successful
0618 22:18:08.553080 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":117,"threads_started":1}
I20260618 22:18:08.554430 4540 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b84b451dbf94bf5b2e4a90b145c09bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b84b451dbf94bf5b2e4a90b145c09bb. Latest consensus state: current_term: 1 leader_uuid: "4b84b451dbf94bf5b2e4a90b145c09bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b84b451dbf94bf5b2e4a90b145c09bb" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38129 } } }
I20260618 22:18:08.554553 4540 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b84b451dbf94bf5b2e4a90b145c09bb [sys.catalog]: This master's current role is: LEADER
I20260618 22:18:08.554901 4552 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:18:08.555128 4539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b84b451dbf94bf5b2e4a90b145c09bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b84b451dbf94bf5b2e4a90b145c09bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b84b451dbf94bf5b2e4a90b145c09bb" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38129 } } }
I20260618 22:18:08.555663 4539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b84b451dbf94bf5b2e4a90b145c09bb [sys.catalog]: This master's current role is: LEADER
I20260618 22:18:08.558375 4552 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:18:08.563329 4552 catalog_manager.cc:1347] Generated new cluster ID: 8bc566b3d0e2458284baae314b8f0f27
I20260618 22:18:08.563390 4552 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:18:08.604477 4552 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:18:08.605764 4552 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:18:08.615278 4552 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b84b451dbf94bf5b2e4a90b145c09bb: Generated new TSK 0
I20260618 22:18:08.616102 4552 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab.
I20260618 22:18:08.648293 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:38129
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:42987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:18:08.754927 4559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:18:08.755187 4559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:18:08.755263 4559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:18:08.759205 4559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:18:08.759292 4559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:18:08.759382 4559 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260618 22:18:08.764454 4559 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:42987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:38129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4559
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:18:08.765404 4559 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:18:08.766374 4559 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:18:08.775110 4567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:08.775208 4559 server_base.cc:1034] running on GCE node
W20260618 22:18:08.775091 4564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:08.775403 4565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:08.775836 4559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:18:08.776433 4559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:18:08.777585 4559 hybrid_clock.cc:648] HybridClock initialized: now 1781821088777559 us; error 41 us; skew 500 ppm
Jun 18 22:18:08 dist-test-slave-g6r4 krb5kdc[4453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821088, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:18:08.780220 4559 init.cc:377] Logged in from keytab as kudu/127.0.98.65@KRBTEST.COM (short username kudu)
I20260618 22:18:08.780967 4559 fs_manager.cc:359] Metadata directory not provided
I20260618 22:18:08.781059 4559 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:18:08.781292 4559 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:18:08.783233 4559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/minicluster-data/ts-0/data/instance:
uuid: "ddabcd5846404b95ad89dbc7041149a6"
format_stamp: "Formatted at 2026-06-18 22:18:08 on dist-test-slave-g6r4"
server_key: "793f51662dd5d44e2376bbf3faa81e44"
server_key_iv: "e155bbb8a33eb04af858e5637a9861bf"
server_key_version: "encryptionkey@0"
I20260618 22:18:08.783784 4559 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/minicluster-data/ts-0/wal/instance:
uuid: "ddabcd5846404b95ad89dbc7041149a6"
format_stamp: "Formatted at 2026-06-18 22:18:08 on dist-test-slave-g6r4"
server_key: "793f51662dd5d44e2376bbf3faa81e44"
server_key_iv: "e155bbb8a33eb04af858e5637a9861bf"
server_key_version: "encryptionkey@0"
I20260618 22:18:08.787375 4559 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:18:08.789567 4573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:08.790789 4559 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:18:08.790908 4559 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/minicluster-data/ts-0/wal
uuid: "ddabcd5846404b95ad89dbc7041149a6"
format_stamp: "Formatted at 2026-06-18 22:18:08 on dist-test-slave-g6r4"
server_key: "793f51662dd5d44e2376bbf3faa81e44"
server_key_iv: "e155bbb8a33eb04af858e5637a9861bf"
server_key_version: "encryptionkey@0"
I20260618 22:18:08.791007 4559 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:18:08.803525 4559 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:18:08.806177 4559 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:18:08.806367 4559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:18:08.806876 4559 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:18:08.807632 4559 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:18:08.807701 4559 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:08.807761 4559 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:18:08.807790 4559 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:08.816097 4559 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:42481
I20260618 22:18:08.816123 4686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:42481 every 8 connection(s)
I20260618 22:18:08.816628 4559 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
I20260618 22:18:08.824667 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 4559
I20260618 22:18:08.824811 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/minicluster-data/ts-0/wal/instance
I20260618 22:18:08.825090 393 external_mini_cluster.cc:1428] Setting key 53157b4c07fffe64095c91d9d082346e
Jun 18 22:18:08 dist-test-slave-g6r4 krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821088, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:18:08.830724 4536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:08.818298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:58877 (local address 127.0.98.126:38129)
0618 22:18:08.818445 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:08.818449 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:08.819123 (+ 674us) server_negotiation.cc:407] Connection header received
0618 22:18:08.820410 (+ 1287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:08.820413 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:08.820458 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:08.820532 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:08.821657 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:08.822264 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:08.823311 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:08.823525 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:08.823600 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:08.826051 (+ 2451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:08.826054 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:08.826056 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:08.826078 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:08.828775 (+ 2697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:08.829275 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:08.829279 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:08.829280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:08.829358 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:08.829686 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:08.829689 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:08.829691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:08.829860 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:08.829999 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:08.830444 (+ 445us) server_negotiation.cc:299] Negotiation successful
0618 22:18:08.830562 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260618 22:18:08.831450 4689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:08.818562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38129 (local address 127.0.98.65:58877)
0618 22:18:08.818981 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:08.819021 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:08.820206 (+ 1185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:08.820683 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:08.820689 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:08.821025 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:08.821535 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:08.821548 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:08.822367 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:08.822370 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:08.823212 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:08.823221 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:08.823652 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:08.823655 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:08.823782 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:08.824296 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:08.824312 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:08.825836 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:18:08.828907 (+ 3071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:08.828912 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:08.828924 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:08.829169 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:08.829451 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:08.829455 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:08.829457 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:08.829586 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:08.829998 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:18:08.830005 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:18:08.830228 (+ 223us) client_negotiation.cc:769] Sending connection context
0618 22:18:08.830405 (+ 177us) client_negotiation.cc:240] Negotiation successful
0618 22:18:08.830689 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":107,"threads_started":1}
I20260618 22:18:08.832754 4687 heartbeater.cc:347] Connected to a master server at 127.0.98.126:38129
I20260618 22:18:08.833007 4687 heartbeater.cc:464] Registering TS with master...
I20260618 22:18:08.833518 4687 heartbeater.cc:511] Master 127.0.98.126:38129 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.66@KRBTEST.COM; defaulting to no policy
I20260618 22:18:08.835078 4497 ts_manager.cc:194] Registered new tserver with Master: ddabcd5846404b95ad89dbc7041149a6 (127.0.98.65:42481)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.66@KRBTEST.COM" created.
I20260618 22:18:08.836338 4497 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.65@KRBTEST.COM'} at 127.0.98.65:58877
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab.
I20260618 22:18:08.851016 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:38129
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:42987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:18:08.965649 4692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:18:08.965888 4692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:18:08.966004 4692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:18:08.969952 4692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:18:08.970010 4692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:18:08.970070 4692 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260618 22:18:08.975086 4692 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:42987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:38129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:18:08.975962 4692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:18:08.976941 4692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:18:08.985975 4700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:08.985980 4697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:08.985980 4698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:08.986384 4692 server_base.cc:1034] running on GCE node
I20260618 22:18:08.986795 4692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:18:08.987558 4692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:18:08.988729 4692 hybrid_clock.cc:648] HybridClock initialized: now 1781821088988718 us; error 24 us; skew 500 ppm
Jun 18 22:18:08 dist-test-slave-g6r4 krb5kdc[4453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821088, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:18:08.991575 4692 init.cc:377] Logged in from keytab as kudu/127.0.98.66@KRBTEST.COM (short username kudu)
I20260618 22:18:08.992296 4692 fs_manager.cc:359] Metadata directory not provided
I20260618 22:18:08.992358 4692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:18:08.992561 4692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:18:08.994549 4692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/minicluster-data/ts-1/data/instance:
uuid: "d0f8f9b5bd3a4cd4a5684f5266a8cac9"
format_stamp: "Formatted at 2026-06-18 22:18:08 on dist-test-slave-g6r4"
server_key: "4e264cf8d1aa4b88aee54e0d2422a915"
server_key_iv: "184966cb2d10c032c67cf4de9699e5cd"
server_key_version: "encryptionkey@0"
I20260618 22:18:08.995096 4692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/minicluster-data/ts-1/wal/instance:
uuid: "d0f8f9b5bd3a4cd4a5684f5266a8cac9"
format_stamp: "Formatted at 2026-06-18 22:18:08 on dist-test-slave-g6r4"
server_key: "4e264cf8d1aa4b88aee54e0d2422a915"
server_key_iv: "184966cb2d10c032c67cf4de9699e5cd"
server_key_version: "encryptionkey@0"
I20260618 22:18:08.998580 4692 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260618 22:18:09.000725 4706 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:09.001860 4692 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260618 22:18:09.002027 4692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/minicluster-data/ts-1/wal
uuid: "d0f8f9b5bd3a4cd4a5684f5266a8cac9"
format_stamp: "Formatted at 2026-06-18 22:18:08 on dist-test-slave-g6r4"
server_key: "4e264cf8d1aa4b88aee54e0d2422a915"
server_key_iv: "184966cb2d10c032c67cf4de9699e5cd"
server_key_version: "encryptionkey@0"
I20260618 22:18:09.002139 4692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:18:09.031803 4692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:18:09.034668 4692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:18:09.034866 4692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:18:09.035410 4692 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:18:09.036186 4692 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:18:09.036231 4692 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:09.036298 4692 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:18:09.036338 4692 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:09.045329 4692 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:32881
I20260618 22:18:09.045347 4819 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:32881 every 8 connection(s)
I20260618 22:18:09.045886 4692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
I20260618 22:18:09.047406 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 4692
I20260618 22:18:09.047537 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/minicluster-data/ts-1/wal/instance
I20260618 22:18:09.047839 393 external_mini_cluster.cc:1428] Setting key 640c66d2fb8061a284cf64270e08833f
Jun 18 22:18:09 dist-test-slave-g6r4 krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821088, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.67@KRBTEST.COM" created.
I20260618 22:18:09.061434 4536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:09.047692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:59681 (local address 127.0.98.126:38129)
0618 22:18:09.047820 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:09.047825 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:09.048725 (+ 900us) server_negotiation.cc:407] Connection header received
0618 22:18:09.050504 (+ 1779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:09.050506 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:09.050547 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:09.050626 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:09.051955 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:09.052826 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:09.054333 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:09.054750 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:09.054900 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:09.057738 (+ 2838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:09.057742 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:09.057745 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:09.057770 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:09.059241 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:09.059703 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:09.059706 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:09.059707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:09.059801 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:09.060199 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:09.060202 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:09.060203 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:09.060391 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:09.060516 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:09.061149 (+ 633us) server_negotiation.cc:299] Negotiation successful
0618 22:18:09.061270 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260618 22:18:09.062410 4822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:09.048116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38129 (local address 127.0.98.66:59681)
0618 22:18:09.048573 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:09.048621 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:09.050282 (+ 1661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:09.050821 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:09.050832 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:09.051321 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:09.051800 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:09.051814 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:09.053001 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:09.053005 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:09.054227 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:09.054237 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:09.054921 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:09.054924 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:09.055112 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:09.055762 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:09.055777 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:09.057565 (+ 1788us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:18:09.059343 (+ 1778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:09.059348 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:09.059360 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:09.059612 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:09.059923 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:09.059925 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:09.059927 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:09.060090 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:09.060536 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:18:09.060543 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:18:09.060867 (+ 324us) client_negotiation.cc:769] Sending connection context
0618 22:18:09.061102 (+ 235us) client_negotiation.cc:240] Negotiation successful
0618 22:18:09.061435 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":124,"threads_started":1}
I20260618 22:18:09.063630 4820 heartbeater.cc:347] Connected to a master server at 127.0.98.126:38129
I20260618 22:18:09.063860 4820 heartbeater.cc:464] Registering TS with master...
I20260618 22:18:09.064311 4820 heartbeater.cc:511] Master 127.0.98.126:38129 requested a full tablet report, sending...
I20260618 22:18:09.065347 4497 ts_manager.cc:194] Registered new tserver with Master: d0f8f9b5bd3a4cd4a5684f5266a8cac9 (127.0.98.66:32881)
I20260618 22:18:09.065963 4497 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.66@KRBTEST.COM'} at 127.0.98.66:59681
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab.
I20260618 22:18:09.076108 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:38129
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:42987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:18:09.184953 4825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:18:09.185181 4825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:18:09.185225 4825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:18:09.189254 4825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:18:09.189307 4825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:18:09.189381 4825 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260618 22:18:09.194681 4825 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:42987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:38129
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:18:09.195626 4825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:18:09.196514 4825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:18:09.205523 4830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:09.205503 4831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:09.205684 4825 server_base.cc:1034] running on GCE node
W20260618 22:18:09.205524 4833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:09.206228 4825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:18:09.206878 4825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:18:09.208056 4825 hybrid_clock.cc:648] HybridClock initialized: now 1781821089208042 us; error 27 us; skew 500 ppm
Jun 18 22:18:09 dist-test-slave-g6r4 krb5kdc[4453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821089, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:18:09.210843 4825 init.cc:377] Logged in from keytab as kudu/127.0.98.67@KRBTEST.COM (short username kudu)
I20260618 22:18:09.211520 4825 fs_manager.cc:359] Metadata directory not provided
I20260618 22:18:09.211614 4825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:18:09.211844 4825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:18:09.213778 4825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/minicluster-data/ts-2/data/instance:
uuid: "9231b10b479141da8ee197b438bb1e28"
format_stamp: "Formatted at 2026-06-18 22:18:09 on dist-test-slave-g6r4"
server_key: "982d4b0bbca8b100adf5277f6da61bea"
server_key_iv: "442d5c62af556f6171da8587a18d0172"
server_key_version: "encryptionkey@0"
I20260618 22:18:09.214411 4825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/minicluster-data/ts-2/wal/instance:
uuid: "9231b10b479141da8ee197b438bb1e28"
format_stamp: "Formatted at 2026-06-18 22:18:09 on dist-test-slave-g6r4"
server_key: "982d4b0bbca8b100adf5277f6da61bea"
server_key_iv: "442d5c62af556f6171da8587a18d0172"
server_key_version: "encryptionkey@0"
I20260618 22:18:09.218001 4825 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:18:09.220309 4839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:09.221609 4825 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:18:09.221741 4825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/minicluster-data/ts-2/wal
uuid: "9231b10b479141da8ee197b438bb1e28"
format_stamp: "Formatted at 2026-06-18 22:18:09 on dist-test-slave-g6r4"
server_key: "982d4b0bbca8b100adf5277f6da61bea"
server_key_iv: "442d5c62af556f6171da8587a18d0172"
server_key_version: "encryptionkey@0"
I20260618 22:18:09.221853 4825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:18:09.240105 4825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:18:09.243438 4825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:18:09.243643 4825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:18:09.244225 4825 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:18:09.245057 4825 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:18:09.245128 4825 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:09.245198 4825 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:18:09.245246 4825 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:09.254088 4825 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:34565
I20260618 22:18:09.254153 4952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:34565 every 8 connection(s)
I20260618 22:18:09.254706 4825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
I20260618 22:18:09.261817 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 4825
I20260618 22:18:09.261969 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/minicluster-data/ts-2/wal/instance
I20260618 22:18:09.262259 393 external_mini_cluster.cc:1428] Setting key b207612196829b2a87df0d55478c31c0
Jun 18 22:18:09 dist-test-slave-g6r4 krb5kdc[4453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821089, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:18:09.268041 4536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:09.256591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:52021 (local address 127.0.98.126:38129)
0618 22:18:09.256744 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:09.256748 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:09.257631 (+ 883us) server_negotiation.cc:407] Connection header received
0618 22:18:09.258996 (+ 1365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:09.258999 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:09.259044 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:09.259122 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:09.260227 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:09.260753 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:09.261848 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:09.262185 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:09.262302 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:09.264863 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:09.264866 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:09.264868 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:09.264889 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:09.266356 (+ 1467us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:09.266825 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:09.266827 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:09.266828 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:09.266896 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:09.267165 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:09.267167 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:09.267168 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:09.267326 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:09.267398 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:09.267844 (+ 446us) server_negotiation.cc:299] Negotiation successful
0618 22:18:09.267932 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260618 22:18:09.268800 4955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:09.257047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38129 (local address 127.0.98.67:52021)
0618 22:18:09.257483 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:09.257525 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:09.258797 (+ 1272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:09.259278 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:09.259295 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:09.259634 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:09.260092 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:09.260105 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:09.260858 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:09.260860 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:09.261711 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:09.261719 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:09.262349 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:09.262353 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:09.262539 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:09.263140 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:09.263156 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:09.264708 (+ 1552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:18:09.266449 (+ 1741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:09.266453 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:09.266464 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:09.266728 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:09.266969 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:09.266971 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:09.266973 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:09.267090 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:09.267407 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:18:09.267412 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:18:09.267613 (+ 201us) client_negotiation.cc:769] Sending connection context
0618 22:18:09.267791 (+ 178us) client_negotiation.cc:240] Negotiation successful
0618 22:18:09.268013 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":100,"threads_started":1}
I20260618 22:18:09.269809 4953 heartbeater.cc:347] Connected to a master server at 127.0.98.126:38129
I20260618 22:18:09.270046 4953 heartbeater.cc:464] Registering TS with master...
I20260618 22:18:09.270409 4953 heartbeater.cc:511] Master 127.0.98.126:38129 requested a full tablet report, sending...
I20260618 22:18:09.271281 4497 ts_manager.cc:194] Registered new tserver with Master: 9231b10b479141da8ee197b438bb1e28 (127.0.98.67:34565)
I20260618 22:18:09.271819 4497 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.67@KRBTEST.COM'} at 127.0.98.67:52021
I20260618 22:18:09.275727 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 22:18:09.285216 4536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:09.277591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50426 (local address 127.0.98.126:38129)
0618 22:18:09.277756 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:09.277760 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:09.277918 (+ 158us) server_negotiation.cc:407] Connection header received
0618 22:18:09.278069 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:09.278072 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:09.278116 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:09.278215 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:09.279096 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:09.279594 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:09.280442 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:09.280653 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:09.280754 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:09.282177 (+ 1423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:09.282180 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:09.282182 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:09.282202 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:09.283433 (+ 1231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:09.283870 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:09.283923 (+ 53us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:09.283929 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:09.284012 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:09.284341 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:09.284349 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:09.284350 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:09.284556 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:09.284690 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:09.284941 (+ 251us) server_negotiation.cc:299] Negotiation successful
0618 22:18:09.285055 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260618 22:18:09.288187 4497 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50426:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 22:18:09.291409 4497 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:18:09.305310 4964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:09.299291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:34565 (local address 127.0.0.1:57642)
0618 22:18:09.299707 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:09.299754 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:09.299875 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:09.300723 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:09.300730 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:09.300758 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:18:09.300896 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:09.300906 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:09.302351 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:09.302356 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:09.303549 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:09.303559 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:09.304650 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:09.304654 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:09.304843 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:09.304888 (+ 45us) client_negotiation.cc:769] Sending connection context
0618 22:18:09.305026 (+ 138us) client_negotiation.cc:240] Negotiation successful
0618 22:18:09.305143 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":28,"thread_start_us":58,"threads_started":1}
I20260618 22:18:09.305596 4965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:09.299308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:32881 (local address 127.0.0.1:56144)
0618 22:18:09.299935 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:09.299949 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:09.300061 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:09.301556 (+ 1495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:09.301559 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:09.301580 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:18:09.301671 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:09.301677 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:09.303174 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:09.303179 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:09.304270 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:09.304279 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:09.305233 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:09.305236 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:09.305316 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:09.305336 (+ 20us) client_negotiation.cc:769] Sending connection context
0618 22:18:09.305396 (+ 60us) client_negotiation.cc:240] Negotiation successful
0618 22:18:09.305464 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"mutex_wait_us":34,"thread_start_us":94,"threads_started":1}
I20260618 22:18:09.305850 4969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:09.300367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56144 (local address 127.0.98.66:32881)
0618 22:18:09.301169 (+ 802us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:09.301179 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:09.301198 (+ 19us) server_negotiation.cc:407] Connection header received
0618 22:18:09.301260 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:09.301264 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:09.301418 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:09.301809 (+ 391us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:18:09.301989 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:09.303046 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:09.304385 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:09.305101 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:09.305235 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:09.305375 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:09.305588 (+ 213us) server_negotiation.cc:299] Negotiation successful
0618 22:18:09.305686 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":697,"thread_start_us":109,"threads_started":1}
I20260618 22:18:09.306118 4967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:09.299963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:42481 (local address 127.0.0.1:38944)
0618 22:18:09.300858 (+ 895us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:09.300871 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:09.300958 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:09.301442 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:09.301445 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:09.301464 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:18:09.301550 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:09.301555 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:09.302957 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:09.302961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:09.304271 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:09.304281 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:09.305686 (+ 1405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:09.305691 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:09.305798 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:09.305819 (+ 21us) client_negotiation.cc:769] Sending connection context
0618 22:18:09.305884 (+ 65us) client_negotiation.cc:240] Negotiation successful
0618 22:18:09.305997 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":833,"mutex_wait_us":134,"thread_start_us":82,"threads_started":1}
I20260618 22:18:09.306363 4968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:09.300319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38944 (local address 127.0.98.65:42481)
0618 22:18:09.300557 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:09.300564 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:09.301049 (+ 485us) server_negotiation.cc:407] Connection header received
0618 22:18:09.301123 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:09.301127 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:09.301300 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:09.301441 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:18:09.301677 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:09.302820 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:09.304409 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:09.305251 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:09.305360 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:09.306069 (+ 709us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:09.306119 (+ 50us) server_negotiation.cc:299] Negotiation successful
0618 22:18:09.306227 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":99,"threads_started":1}
I20260618 22:18:09.306591 4966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:09.299405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57642 (local address 127.0.98.67:34565)
0618 22:18:09.300036 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:09.300048 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:09.300070 (+ 22us) server_negotiation.cc:407] Connection header received
0618 22:18:09.300158 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:09.300164 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:09.300381 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:09.300544 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:18:09.301003 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:09.302218 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:09.303686 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:09.304525 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:09.305587 (+ 1062us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:09.306117 (+ 530us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:09.306322 (+ 205us) server_negotiation.cc:299] Negotiation successful
0618 22:18:09.306397 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":146,"threads_started":1}
I20260618 22:18:09.308879 4887 tablet_service.cc:1467] Processing CreateTablet for tablet d991ea6436aa4e4e9011e7635106f5af (DEFAULT_TABLE table=test-table [id=88bec73b52af4a849942a8751ed44d2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:18:09.308769 4754 tablet_service.cc:1467] Processing CreateTablet for tablet d991ea6436aa4e4e9011e7635106f5af (DEFAULT_TABLE table=test-table [id=88bec73b52af4a849942a8751ed44d2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:18:09.309145 4621 tablet_service.cc:1467] Processing CreateTablet for tablet d991ea6436aa4e4e9011e7635106f5af (DEFAULT_TABLE table=test-table [id=88bec73b52af4a849942a8751ed44d2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:18:09.309791 4887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d991ea6436aa4e4e9011e7635106f5af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:18:09.309808 4754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d991ea6436aa4e4e9011e7635106f5af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:18:09.310011 4621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d991ea6436aa4e4e9011e7635106f5af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:18:09.316044 4970 tablet_bootstrap.cc:492] T d991ea6436aa4e4e9011e7635106f5af P 9231b10b479141da8ee197b438bb1e28: Bootstrap starting.
I20260618 22:18:09.316529 4971 tablet_bootstrap.cc:492] T d991ea6436aa4e4e9011e7635106f5af P d0f8f9b5bd3a4cd4a5684f5266a8cac9: Bootstrap starting.
I20260618 22:18:09.317731 4972 tablet_bootstrap.cc:492] T d991ea6436aa4e4e9011e7635106f5af P ddabcd5846404b95ad89dbc7041149a6: Bootstrap starting.
I20260618 22:18:09.318444 4970 tablet_bootstrap.cc:654] T d991ea6436aa4e4e9011e7635106f5af P 9231b10b479141da8ee197b438bb1e28: Neither blocks nor log segments found. Creating new log.
I20260618 22:18:09.319125 4971 tablet_bootstrap.cc:654] T d991ea6436aa4e4e9011e7635106f5af P d0f8f9b5bd3a4cd4a5684f5266a8cac9: Neither blocks nor log segments found. Creating new log.
I20260618 22:18:09.319536 4972 tablet_bootstrap.cc:654] T d991ea6436aa4e4e9011e7635106f5af P ddabcd5846404b95ad89dbc7041149a6: Neither blocks nor log segments found. Creating new log.
I20260618 22:18:09.319761 4970 log.cc:824] T d991ea6436aa4e4e9011e7635106f5af P 9231b10b479141da8ee197b438bb1e28: Log is configured to *not* fsync() on all Append() calls
I20260618 22:18:09.319811 4971 log.cc:824] T d991ea6436aa4e4e9011e7635106f5af P d0f8f9b5bd3a4cd4a5684f5266a8cac9: Log is configured to *not* fsync() on all Append() calls
I20260618 22:18:09.320770 4972 log.cc:824] T d991ea6436aa4e4e9011e7635106f5af P ddabcd5846404b95ad89dbc7041149a6: Log is configured to *not* fsync() on all Append() calls
I20260618 22:18:09.321674 4971 tablet_bootstrap.cc:492] T d991ea6436aa4e4e9011e7635106f5af P d0f8f9b5bd3a4cd4a5684f5266a8cac9: No bootstrap required, opened a new log
I20260618 22:18:09.321863 4971 ts_tablet_manager.cc:1397] T d991ea6436aa4e4e9011e7635106f5af P d0f8f9b5bd3a4cd4a5684f5266a8cac9: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260618 22:18:09.322477 4972 tablet_bootstrap.cc:492] T d991ea6436aa4e4e9011e7635106f5af P ddabcd5846404b95ad89dbc7041149a6: No bootstrap required, opened a new log
I20260618 22:18:09.322664 4972 ts_tablet_manager.cc:1397] T d991ea6436aa4e4e9011e7635106f5af P ddabcd5846404b95ad89dbc7041149a6: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260618 22:18:09.324620 4971 raft_consensus.cc:348] T d991ea6436aa4e4e9011e7635106f5af P d0f8f9b5bd3a4cd4a5684f5266a8cac9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9231b10b479141da8ee197b438bb1e28" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34565 } } peers { permanent_uuid: "d0f8f9b5bd3a4cd4a5684f5266a8cac9" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 32881 } } peers { permanent_uuid: "ddabcd5846404b95ad89dbc7041149a6" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 42481 } }
I20260618 22:18:09.324832 4971 raft_consensus.cc:374] T d991ea6436aa4e4e9011e7635106f5af P d0f8f9b5bd3a4cd4a5684f5266a8cac9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:18:09.324891 4971 raft_consensus.cc:729] T d991ea6436aa4e4e9011e7635106f5af P d0f8f9b5bd3a4cd4a5684f5266a8cac9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0f8f9b5bd3a4cd4a5684f5266a8cac9, State: Initialized, Role: FOLLOWER
I20260618 22:18:09.325336 4971 consensus_queue.cc:260] T d991ea6436aa4e4e9011e7635106f5af P d0f8f9b5bd3a4cd4a5684f5266a8cac9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9231b10b479141da8ee197b438bb1e28" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34565 } } peers { permanent_uuid: "d0f8f9b5bd3a4cd4a5684f5266a8cac9" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 32881 } } peers { permanent_uuid: "ddabcd5846404b95ad89dbc7041149a6" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 42481 } }
I20260618 22:18:09.325562 4972 raft_consensus.cc:348] T d991ea6436aa4e4e9011e7635106f5af P ddabcd5846404b95ad89dbc7041149a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9231b10b479141da8ee197b438bb1e28" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34565 } } peers { permanent_uuid: "d0f8f9b5bd3a4cd4a5684f5266a8cac9" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 32881 } } peers { permanent_uuid: "ddabcd5846404b95ad89dbc7041149a6" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 42481 } }
I20260618 22:18:09.325819 4972 raft_consensus.cc:374] T d991ea6436aa4e4e9011e7635106f5af P ddabcd5846404b95ad89dbc7041149a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:18:09.325881 4972 raft_consensus.cc:729] T d991ea6436aa4e4e9011e7635106f5af P ddabcd5846404b95ad89dbc7041149a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ddabcd5846404b95ad89dbc7041149a6, State: Initialized, Role: FOLLOWER
I20260618 22:18:09.326131 4820 heartbeater.cc:503] Master 127.0.98.126:38129 was elected leader, sending a full tablet report...
I20260618 22:18:09.326167 4970 tablet_bootstrap.cc:492] T d991ea6436aa4e4e9011e7635106f5af P 9231b10b479141da8ee197b438bb1e28: No bootstrap required, opened a new log
I20260618 22:18:09.326304 4971 ts_tablet_manager.cc:1428] T d991ea6436aa4e4e9011e7635106f5af P d0f8f9b5bd3a4cd4a5684f5266a8cac9: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
I20260618 22:18:09.326323 4970 ts_tablet_manager.cc:1397] T d991ea6436aa4e4e9011e7635106f5af P 9231b10b479141da8ee197b438bb1e28: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260618 22:18:09.326366 4972 consensus_queue.cc:260] T d991ea6436aa4e4e9011e7635106f5af P ddabcd5846404b95ad89dbc7041149a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9231b10b479141da8ee197b438bb1e28" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34565 } } peers { permanent_uuid: "d0f8f9b5bd3a4cd4a5684f5266a8cac9" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 32881 } } peers { permanent_uuid: "ddabcd5846404b95ad89dbc7041149a6" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 42481 } }
I20260618 22:18:09.327018 4687 heartbeater.cc:503] Master 127.0.98.126:38129 was elected leader, sending a full tablet report...
I20260618 22:18:09.327515 4972 ts_tablet_manager.cc:1428] T d991ea6436aa4e4e9011e7635106f5af P ddabcd5846404b95ad89dbc7041149a6: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260618 22:18:09.329698 4970 raft_consensus.cc:348] T d991ea6436aa4e4e9011e7635106f5af P 9231b10b479141da8ee197b438bb1e28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9231b10b479141da8ee197b438bb1e28" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34565 } } peers { permanent_uuid: "d0f8f9b5bd3a4cd4a5684f5266a8cac9" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 32881 } } peers { permanent_uuid: "ddabcd5846404b95ad89dbc7041149a6" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 42481 } }
I20260618 22:18:09.329988 4970 raft_consensus.cc:374] T d991ea6436aa4e4e9011e7635106f5af P 9231b10b479141da8ee197b438bb1e28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:18:09.330080 4970 raft_consensus.cc:729] T d991ea6436aa4e4e9011e7635106f5af P 9231b10b479141da8ee197b438bb1e28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9231b10b479141da8ee197b438bb1e28, State: Initialized, Role: FOLLOWER
I20260618 22:18:09.330454 4970 consensus_queue.cc:260] T d991ea6436aa4e4e9011e7635106f5af P 9231b10b479141da8ee197b438bb1e28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9231b10b479141da8ee197b438bb1e28" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34565 } } peers { permanent_uuid: "d0f8f9b5bd3a4cd4a5684f5266a8cac9" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 32881 } } peers { permanent_uuid: "ddabcd5846404b95ad89dbc7041149a6" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 42481 } }
I20260618 22:18:09.331073 4953 heartbeater.cc:503] Master 127.0.98.126:38129 was elected leader, sending a full tablet report...
I20260618 22:18:09.331362 4970 ts_tablet_manager.cc:1428] T d991ea6436aa4e4e9011e7635106f5af P 9231b10b479141da8ee197b438bb1e28: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260618 22:18:09.505640 4954 tablet.cc:2369] T d991ea6436aa4e4e9011e7635106f5af P 9231b10b479141da8ee197b438bb1e28: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260618 22:18:09.546900 4821 tablet.cc:2369] T d991ea6436aa4e4e9011e7635106f5af P d0f8f9b5bd3a4cd4a5684f5266a8cac9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260618 22:18:09.567780 4688 tablet.cc:2369] T d991ea6436aa4e4e9011e7635106f5af P ddabcd5846404b95ad89dbc7041149a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:18:09.634979 4978 raft_consensus.cc:482] T d991ea6436aa4e4e9011e7635106f5af P 9231b10b479141da8ee197b438bb1e28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:18:09.635193 4978 raft_consensus.cc:504] T d991ea6436aa4e4e9011e7635106f5af P 9231b10b479141da8ee197b438bb1e28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9231b10b479141da8ee197b438bb1e28" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34565 } } peers { permanent_uuid: "d0f8f9b5bd3a4cd4a5684f5266a8cac9" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 32881 } } peers { permanent_uuid: "ddabcd5846404b95ad89dbc7041149a6" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 42481 } }
I20260618 22:18:09.636421 4978 leader_election.cc:288] T d991ea6436aa4e4e9011e7635106f5af P 9231b10b479141da8ee197b438bb1e28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d0f8f9b5bd3a4cd4a5684f5266a8cac9 (127.0.98.66:32881), ddabcd5846404b95ad89dbc7041149a6 (127.0.98.65:42481)
I20260618 22:18:09.640133 4969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:09.636834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:45483 (local address 127.0.98.66:32881)
0618 22:18:09.636972 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:09.636977 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:09.636995 (+ 18us) server_negotiation.cc:407] Connection header received
0618 22:18:09.637115 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:09.637121 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:09.637177 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:09.637271 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:18:09.637549 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:09.638103 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:09.639190 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:09.639585 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:09.639705 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:09.639768 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:09.639958 (+ 190us) server_negotiation.cc:299] Negotiation successful
0618 22:18:09.640023 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260618 22:18:09.640134 4955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:09.636685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:32881 (local address 127.0.98.67:45483)
0618 22:18:09.636842 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:09.636856 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:09.636992 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:09.637299 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:09.637302 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:09.637318 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:18:09.637436 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:09.637441 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:09.638209 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:09.638215 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:09.639064 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:09.639073 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:09.639703 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:09.639707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:09.639816 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:09.639844 (+ 28us) client_negotiation.cc:769] Sending connection context
0618 22:18:09.639928 (+ 84us) client_negotiation.cc:240] Negotiation successful
0618 22:18:09.640001 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63,"mutex_wait_us":22}
I20260618 22:18:09.640561 4968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:09.637135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:60237 (local address 127.0.98.65:42481)
0618 22:18:09.637269 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:09.637274 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:09.637412 (+ 138us) server_negotiation.cc:407] Connection header received
0618 22:18:09.637560 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:09.637564 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:09.637626 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:09.637711 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:18:09.638002 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:09.638490 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:09.639573 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:09.639975 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:09.640094 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:09.640150 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:09.640378 (+ 228us) server_negotiation.cc:299] Negotiation successful
0618 22:18:09.640441 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260618 22:18:09.640784 4774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d991ea6436aa4e4e9011e7635106f5af" candidate_uuid: "9231b10b479141da8ee197b438bb1e28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0f8f9b5bd3a4cd4a5684f5266a8cac9" is_pre_election: true
I20260618 22:18:09.640923 4979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:09.636979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:42481 (local address 127.0.98.67:60237)
0618 22:18:09.637314 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:09.637334 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:09.637447 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:09.637721 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:09.637724 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:09.637738 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:18:09.637856 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:09.637861 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:09.638602 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:09.638606 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:09.639443 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:09.639452 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:09.640094 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:09.640105 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:09.640206 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:09.640238 (+ 32us) client_negotiation.cc:769] Sending connection context
0618 22:18:09.640733 (+ 495us) client_negotiation.cc:240] Negotiation successful
0618 22:18:09.640803 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"mutex_wait_us":20,"thread_start_us":114,"threads_started":1}
I20260618 22:18:09.641115 4774 raft_consensus.cc:2454] T d991ea6436aa4e4e9011e7635106f5af P d0f8f9b5bd3a4cd4a5684f5266a8cac9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9231b10b479141da8ee197b438bb1e28 in term 0.
I20260618 22:18:09.641566 4641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d991ea6436aa4e4e9011e7635106f5af" candidate_uuid: "9231b10b479141da8ee197b438bb1e28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddabcd5846404b95ad89dbc7041149a6" is_pre_election: true
I20260618 22:18:09.641615 4843 leader_election.cc:302] T d991ea6436aa4e4e9011e7635106f5af P 9231b10b479141da8ee197b438bb1e28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9231b10b479141da8ee197b438bb1e28, d0f8f9b5bd3a4cd4a5684f5266a8cac9; no voters:
I20260618 22:18:09.641862 4641 raft_consensus.cc:2454] T d991ea6436aa4e4e9011e7635106f5af P ddabcd5846404b95ad89dbc7041149a6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9231b10b479141da8ee197b438bb1e28 in term 0.
I20260618 22:18:09.641927 4978 raft_consensus.cc:2781] T d991ea6436aa4e4e9011e7635106f5af P 9231b10b479141da8ee197b438bb1e28 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:18:09.642019 4978 raft_consensus.cc:482] T d991ea6436aa4e4e9011e7635106f5af P 9231b10b479141da8ee197b438bb1e28 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:18:09.642073 4978 raft_consensus.cc:3037] T d991ea6436aa4e4e9011e7635106f5af P 9231b10b479141da8ee197b438bb1e28 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:18:09.643064 4978 raft_consensus.cc:504] T d991ea6436aa4e4e9011e7635106f5af P 9231b10b479141da8ee197b438bb1e28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9231b10b479141da8ee197b438bb1e28" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34565 } } peers { permanent_uuid: "d0f8f9b5bd3a4cd4a5684f5266a8cac9" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 32881 } } peers { permanent_uuid: "ddabcd5846404b95ad89dbc7041149a6" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 42481 } }
I20260618 22:18:09.643343 4978 leader_election.cc:288] T d991ea6436aa4e4e9011e7635106f5af P 9231b10b479141da8ee197b438bb1e28 [CANDIDATE]: Term 1 election: Requested vote from peers d0f8f9b5bd3a4cd4a5684f5266a8cac9 (127.0.98.66:32881), ddabcd5846404b95ad89dbc7041149a6 (127.0.98.65:42481)
I20260618 22:18:09.643745 4774 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d991ea6436aa4e4e9011e7635106f5af" candidate_uuid: "9231b10b479141da8ee197b438bb1e28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0f8f9b5bd3a4cd4a5684f5266a8cac9"
I20260618 22:18:09.643848 4641 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d991ea6436aa4e4e9011e7635106f5af" candidate_uuid: "9231b10b479141da8ee197b438bb1e28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ddabcd5846404b95ad89dbc7041149a6"
I20260618 22:18:09.643852 4774 raft_consensus.cc:3037] T d991ea6436aa4e4e9011e7635106f5af P d0f8f9b5bd3a4cd4a5684f5266a8cac9 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:18:09.644001 4641 raft_consensus.cc:3037] T d991ea6436aa4e4e9011e7635106f5af P ddabcd5846404b95ad89dbc7041149a6 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:18:09.644891 4774 raft_consensus.cc:2454] T d991ea6436aa4e4e9011e7635106f5af P d0f8f9b5bd3a4cd4a5684f5266a8cac9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9231b10b479141da8ee197b438bb1e28 in term 1.
I20260618 22:18:09.645110 4641 raft_consensus.cc:2454] T d991ea6436aa4e4e9011e7635106f5af P ddabcd5846404b95ad89dbc7041149a6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9231b10b479141da8ee197b438bb1e28 in term 1.
I20260618 22:18:09.645190 4843 leader_election.cc:302] T d991ea6436aa4e4e9011e7635106f5af P 9231b10b479141da8ee197b438bb1e28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9231b10b479141da8ee197b438bb1e28, d0f8f9b5bd3a4cd4a5684f5266a8cac9; no voters:
I20260618 22:18:09.645407 4978 raft_consensus.cc:2781] T d991ea6436aa4e4e9011e7635106f5af P 9231b10b479141da8ee197b438bb1e28 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:18:09.645628 4978 raft_consensus.cc:686] T d991ea6436aa4e4e9011e7635106f5af P 9231b10b479141da8ee197b438bb1e28 [term 1 LEADER]: Becoming Leader. State: Replica: 9231b10b479141da8ee197b438bb1e28, State: Running, Role: LEADER
I20260618 22:18:09.645982 4978 consensus_queue.cc:237] T d991ea6436aa4e4e9011e7635106f5af P 9231b10b479141da8ee197b438bb1e28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9231b10b479141da8ee197b438bb1e28" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34565 } } peers { permanent_uuid: "d0f8f9b5bd3a4cd4a5684f5266a8cac9" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 32881 } } peers { permanent_uuid: "ddabcd5846404b95ad89dbc7041149a6" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 42481 } }
I20260618 22:18:09.649039 4496 catalog_manager.cc:5495] T d991ea6436aa4e4e9011e7635106f5af P 9231b10b479141da8ee197b438bb1e28 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9231b10b479141da8ee197b438bb1e28 (127.0.98.67). New cstate: current_term: 1 leader_uuid: "9231b10b479141da8ee197b438bb1e28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9231b10b479141da8ee197b438bb1e28" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 34565 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d0f8f9b5bd3a4cd4a5684f5266a8cac9" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 32881 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ddabcd5846404b95ad89dbc7041149a6" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 42481 } health_report { overall_health: UNKNOWN } } }
I20260618 22:18:09.665673 4966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:09.662367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57652 (local address 127.0.98.67:34565)
0618 22:18:09.662489 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:09.662492 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:09.662506 (+ 14us) server_negotiation.cc:407] Connection header received
0618 22:18:09.662644 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:09.662646 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:09.662696 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:09.662779 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:18:09.662998 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:09.663506 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:09.664313 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:09.664523 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:09.664609 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:09.664869 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:18:09.665283 (+ 414us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:18:09.665374 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:09.665430 (+ 56us) server_negotiation.cc:299] Negotiation successful
0618 22:18:09.665474 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260618 22:18:09.671088 4774 raft_consensus.cc:1261] T d991ea6436aa4e4e9011e7635106f5af P d0f8f9b5bd3a4cd4a5684f5266a8cac9 [term 1 FOLLOWER]: Refusing update from remote peer 9231b10b479141da8ee197b438bb1e28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:18:09.671110 4641 raft_consensus.cc:1261] T d991ea6436aa4e4e9011e7635106f5af P ddabcd5846404b95ad89dbc7041149a6 [term 1 FOLLOWER]: Refusing update from remote peer 9231b10b479141da8ee197b438bb1e28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:18:09.671803 4980 consensus_queue.cc:1035] T d991ea6436aa4e4e9011e7635106f5af P 9231b10b479141da8ee197b438bb1e28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ddabcd5846404b95ad89dbc7041149a6" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 42481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:18:09.671972 4978 consensus_queue.cc:1035] T d991ea6436aa4e4e9011e7635106f5af P 9231b10b479141da8ee197b438bb1e28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0f8f9b5bd3a4cd4a5684f5266a8cac9" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 32881 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:18:09.679394 4984 mvcc.cc:205] Tried to move back new op lower bound from 7298339183285821440 to 7298339183192477696. Current Snapshot: MvccSnapshot[applied={T|T < 7298339183285821440}]
I20260618 22:18:09.687516 4496 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50426:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:18:09.687651 4496 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50426:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:18:09.689855 4496 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4b84b451dbf94bf5b2e4a90b145c09bb: Sending DeleteTablet for 3 replicas of tablet d991ea6436aa4e4e9011e7635106f5af
I20260618 22:18:09.690568 4887 tablet_service.cc:1514] Processing DeleteTablet for tablet d991ea6436aa4e4e9011e7635106f5af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:18:09 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:57642
I20260618 22:18:09.690709 4621 tablet_service.cc:1514] Processing DeleteTablet for tablet d991ea6436aa4e4e9011e7635106f5af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:18:09 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:38944
I20260618 22:18:09.690975 4993 tablet_replica.cc:330] T d991ea6436aa4e4e9011e7635106f5af P 9231b10b479141da8ee197b438bb1e28: stopping tablet replica
I20260618 22:18:09.691195 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 4559
I20260618 22:18:09.691282 4993 raft_consensus.cc:2229] T d991ea6436aa4e4e9011e7635106f5af P 9231b10b479141da8ee197b438bb1e28 [term 1 LEADER]: Raft consensus shutting down.
I20260618 22:18:09.691409 4754 tablet_service.cc:1514] Processing DeleteTablet for tablet d991ea6436aa4e4e9011e7635106f5af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:18:09 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:56144
I20260618 22:18:09.691728 4993 raft_consensus.cc:2258] T d991ea6436aa4e4e9011e7635106f5af P 9231b10b479141da8ee197b438bb1e28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:18:09.693034 4993 ts_tablet_manager.cc:1905] T d991ea6436aa4e4e9011e7635106f5af P 9231b10b479141da8ee197b438bb1e28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:18:09.695787 4993 ts_tablet_manager.cc:1918] T d991ea6436aa4e4e9011e7635106f5af P 9231b10b479141da8ee197b438bb1e28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:18:09.695896 4993 log.cc:1196] T d991ea6436aa4e4e9011e7635106f5af P 9231b10b479141da8ee197b438bb1e28: Deleting WAL directory at /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781821065380026-393-0/minicluster-data/ts-2/wal/wals/d991ea6436aa4e4e9011e7635106f5af
I20260618 22:18:09.696192 4993 ts_tablet_manager.cc:1939] T d991ea6436aa4e4e9011e7635106f5af P 9231b10b479141da8ee197b438bb1e28: Deleting consensus metadata
I20260618 22:18:09.697150 4481 catalog_manager.cc:4841] TS 9231b10b479141da8ee197b438bb1e28 (127.0.98.67:34565): tablet d991ea6436aa4e4e9011e7635106f5af (table test-table [id=88bec73b52af4a849942a8751ed44d2d]) successfully deleted
I20260618 22:18:09.698091 4995 tablet_replica.cc:330] T d991ea6436aa4e4e9011e7635106f5af P d0f8f9b5bd3a4cd4a5684f5266a8cac9: stopping tablet replica
I20260618 22:18:09.698391 4995 raft_consensus.cc:2229] T d991ea6436aa4e4e9011e7635106f5af P d0f8f9b5bd3a4cd4a5684f5266a8cac9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:18:09.698733 4995 raft_consensus.cc:2258] T d991ea6436aa4e4e9011e7635106f5af P d0f8f9b5bd3a4cd4a5684f5266a8cac9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:18:09.699921 4995 ts_tablet_manager.cc:1905] T d991ea6436aa4e4e9011e7635106f5af P d0f8f9b5bd3a4cd4a5684f5266a8cac9: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260618 22:18:09.700434 4484 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.65:42481 (error 108)
I20260618 22:18:09.701381 4967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:09.701071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:42481 (local address 127.0.0.1:38956)
0618 22:18:09.701212 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:09.701281 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.65:42481: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260618 22:18:09.701406 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 4692
W20260618 22:18:09.701638 4484 catalog_manager.cc:4584] TS ddabcd5846404b95ad89dbc7041149a6 (127.0.98.65:42481): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d991ea6436aa4e4e9011e7635106f5af: Network error: Client connection negotiation failed: client connection to 127.0.98.65:42481: connect: Connection refused (error 111)
I20260618 22:18:09.708880 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 4825
I20260618 22:18:09.709113 4967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:09.708880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:32881 (local address 127.0.0.1:56146)
0618 22:18:09.709000 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:09.709048 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.66:32881: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260618 22:18:09.714210 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 4466
2026-06-18T22:18:09Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2543 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-18T22:18:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:18:09Z Disabled control of system clock
I20260618 22:18:09.743085 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821065380026-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821065380026-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821065380026-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:39187
--webserver_interface=127.0.98.126
--webserver_port=0
--builtin_ntp_servers=127.0.98.84:44323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:39187
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260618 22:18:09.848452 5000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:18:09.848685 5000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:18:09.848727 5000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:18:09.852623 5000 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:18:09.852672 5000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:18:09.852687 5000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:18:09.852703 5000 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:18:09.852723 5000 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:18:09.857208 5000 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:44323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821065380026-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821065380026-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:39187
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:39187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821065380026-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:18:09.858196 5000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:18:09.859454 5000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:18:09.867177 5006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:09.867290 5000 server_base.cc:1034] running on GCE node
W20260618 22:18:09.867185 5005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:09.867507 5008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:09.868086 5000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:18:09.869087 5000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:18:09.870275 5000 hybrid_clock.cc:648] HybridClock initialized: now 1781821089870263 us; error 25 us; skew 500 ppm
I20260618 22:18:09.871973 5000 webserver.cc:460] Webserver started at http://127.0.98.126:36323/ using document root <none> and password file <none>
I20260618 22:18:09.872526 5000 fs_manager.cc:359] Metadata directory not provided
I20260618 22:18:09.872619 5000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:18:09.872843 5000 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:18:09.874817 5000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821065380026-393-0/minicluster-data/master-0/data/instance:
uuid: "1f4580ccda1b4813abdc927a4fce0d8f"
format_stamp: "Formatted at 2026-06-18 22:18:09 on dist-test-slave-g6r4"
server_key: "c7133653b0324e30f1b4158204b14962"
server_key_iv: "33168eb47060b29290ac7f6cd0b6e057"
server_key_version: "encryptionkey@0"
I20260618 22:18:09.875365 5000 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821065380026-393-0/minicluster-data/master-0/wal/instance:
uuid: "1f4580ccda1b4813abdc927a4fce0d8f"
format_stamp: "Formatted at 2026-06-18 22:18:09 on dist-test-slave-g6r4"
server_key: "c7133653b0324e30f1b4158204b14962"
server_key_iv: "33168eb47060b29290ac7f6cd0b6e057"
server_key_version: "encryptionkey@0"
I20260618 22:18:09.878898 5000 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.001s
I20260618 22:18:09.881110 5014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:09.882360 5000 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:18:09.882491 5000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821065380026-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821065380026-393-0/minicluster-data/master-0/wal
uuid: "1f4580ccda1b4813abdc927a4fce0d8f"
format_stamp: "Formatted at 2026-06-18 22:18:09 on dist-test-slave-g6r4"
server_key: "c7133653b0324e30f1b4158204b14962"
server_key_iv: "33168eb47060b29290ac7f6cd0b6e057"
server_key_version: "encryptionkey@0"
I20260618 22:18:09.882597 5000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821065380026-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821065380026-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821065380026-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:18:09.886886 5000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:18:09.887473 5000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:18:09.887678 5000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:18:09.894361 5000 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:39187
I20260618 22:18:09.894369 5066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:39187 every 8 connection(s)
I20260618 22:18:09.895202 5000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
I20260618 22:18:09.897835 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 5000
I20260618 22:18:09.897960 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821065380026-393-0/minicluster-data/master-0/wal/instance
I20260618 22:18:09.898267 393 external_mini_cluster.cc:1428] Setting key ed391c799a18641adb9e3fa82e9b6348
I20260618 22:18:09.898361 5067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:18:09.905159 5067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f4580ccda1b4813abdc927a4fce0d8f: Bootstrap starting.
I20260618 22:18:09.906443 5070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:09.900272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49582 (local address 127.0.98.126:39187)
0618 22:18:09.901066 (+ 794us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:09.901089 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:09.901133 (+ 44us) server_negotiation.cc:407] Connection header received
0618 22:18:09.902342 (+ 1209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:09.902370 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:09.902900 (+ 530us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:09.903436 (+ 536us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:09.904468 (+ 1032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:09.904475 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:09.904534 (+ 59us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:18:09.904577 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:09.904712 (+ 135us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:18:09.904743 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:09.904897 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:09.905079 (+ 182us) server_negotiation.cc:299] Negotiation successful
0618 22:18:09.905334 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":121,"threads_started":1}
I20260618 22:18:09.907986 5067 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f4580ccda1b4813abdc927a4fce0d8f: Neither blocks nor log segments found. Creating new log.
I20260618 22:18:09.908807 5067 log.cc:824] T 00000000000000000000000000000000 P 1f4580ccda1b4813abdc927a4fce0d8f: Log is configured to *not* fsync() on all Append() calls
I20260618 22:18:09.910938 5067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f4580ccda1b4813abdc927a4fce0d8f: No bootstrap required, opened a new log
I20260618 22:18:09.915266 5067 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f4580ccda1b4813abdc927a4fce0d8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f4580ccda1b4813abdc927a4fce0d8f" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39187 } }
I20260618 22:18:09.915566 5067 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f4580ccda1b4813abdc927a4fce0d8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:18:09.915627 5067 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f4580ccda1b4813abdc927a4fce0d8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f4580ccda1b4813abdc927a4fce0d8f, State: Initialized, Role: FOLLOWER
I20260618 22:18:09.916157 5067 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f4580ccda1b4813abdc927a4fce0d8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f4580ccda1b4813abdc927a4fce0d8f" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39187 } }
I20260618 22:18:09.916343 5067 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f4580ccda1b4813abdc927a4fce0d8f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:18:09.916422 5067 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f4580ccda1b4813abdc927a4fce0d8f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:18:09.916512 5067 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f4580ccda1b4813abdc927a4fce0d8f [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:18:09.917690 5067 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f4580ccda1b4813abdc927a4fce0d8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f4580ccda1b4813abdc927a4fce0d8f" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39187 } }
I20260618 22:18:09.918068 5067 leader_election.cc:302] T 00000000000000000000000000000000 P 1f4580ccda1b4813abdc927a4fce0d8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f4580ccda1b4813abdc927a4fce0d8f; no voters:
I20260618 22:18:09.918457 5067 leader_election.cc:288] T 00000000000000000000000000000000 P 1f4580ccda1b4813abdc927a4fce0d8f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:18:09.918560 5072 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f4580ccda1b4813abdc927a4fce0d8f [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:18:09.918777 5072 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f4580ccda1b4813abdc927a4fce0d8f [term 1 LEADER]: Becoming Leader. State: Replica: 1f4580ccda1b4813abdc927a4fce0d8f, State: Running, Role: LEADER
I20260618 22:18:09.919111 5072 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f4580ccda1b4813abdc927a4fce0d8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f4580ccda1b4813abdc927a4fce0d8f" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39187 } }
I20260618 22:18:09.919584 5067 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f4580ccda1b4813abdc927a4fce0d8f [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:18:09.920447 5072 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f4580ccda1b4813abdc927a4fce0d8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f4580ccda1b4813abdc927a4fce0d8f. Latest consensus state: current_term: 1 leader_uuid: "1f4580ccda1b4813abdc927a4fce0d8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f4580ccda1b4813abdc927a4fce0d8f" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39187 } } }
I20260618 22:18:09.920569 5072 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f4580ccda1b4813abdc927a4fce0d8f [sys.catalog]: This master's current role is: LEADER
I20260618 22:18:09.920838 5073 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f4580ccda1b4813abdc927a4fce0d8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f4580ccda1b4813abdc927a4fce0d8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f4580ccda1b4813abdc927a4fce0d8f" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39187 } } }
I20260618 22:18:09.920922 5073 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f4580ccda1b4813abdc927a4fce0d8f [sys.catalog]: This master's current role is: LEADER
I20260618 22:18:09.920913 5081 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:18:09.924304 5081 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:18:09.930037 5081 catalog_manager.cc:1347] Generated new cluster ID: a4d00deae2de4bd9b53cc43beef7e83a
I20260618 22:18:09.930100 5081 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:18:09.942862 5081 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:18:09.943214 5081 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:18:09.950953 5081 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f4580ccda1b4813abdc927a4fce0d8f: Generated new TSK 0
I20260618 22:18:09.951684 5081 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:18:09.957370 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821065380026-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821065380026-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821065380026-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:39187
--builtin_ntp_servers=127.0.98.84:44323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260618 22:18:10.064152 5091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:18:10.064405 5091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:18:10.064486 5091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:18:10.068491 5091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:18:10.068569 5091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:18:10.068652 5091 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260618 22:18:10.073299 5091 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:44323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821065380026-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821065380026-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:39187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821065380026-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:18:10.074375 5091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:18:10.075701 5091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:18:10.084952 5099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:10.085013 5097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:10.085105 5091 server_base.cc:1034] running on GCE node
W20260618 22:18:10.084952 5096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:10.085647 5091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:18:10.086387 5091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:18:10.087563 5091 hybrid_clock.cc:648] HybridClock initialized: now 1781821090087532 us; error 44 us; skew 500 ppm
I20260618 22:18:10.089270 5091 webserver.cc:460] Webserver started at http://127.0.98.65:45589/ using document root <none> and password file <none>
I20260618 22:18:10.089838 5091 fs_manager.cc:359] Metadata directory not provided
I20260618 22:18:10.089958 5091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:18:10.090178 5091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:18:10.092144 5091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821065380026-393-0/minicluster-data/ts-0/data/instance:
uuid: "c885a6d32c3e435ab642c24eb24d91bd"
format_stamp: "Formatted at 2026-06-18 22:18:10 on dist-test-slave-g6r4"
server_key: "c94de0de04736a6a9ce18d8417cf77fd"
server_key_iv: "fee46e7fc8f82929cff783cb98dbcbf3"
server_key_version: "encryptionkey@0"
I20260618 22:18:10.092720 5091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821065380026-393-0/minicluster-data/ts-0/wal/instance:
uuid: "c885a6d32c3e435ab642c24eb24d91bd"
format_stamp: "Formatted at 2026-06-18 22:18:10 on dist-test-slave-g6r4"
server_key: "c94de0de04736a6a9ce18d8417cf77fd"
server_key_iv: "fee46e7fc8f82929cff783cb98dbcbf3"
server_key_version: "encryptionkey@0"
I20260618 22:18:10.096272 5091 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260618 22:18:10.098520 5105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:10.099668 5091 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:18:10.099799 5091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821065380026-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821065380026-393-0/minicluster-data/ts-0/wal
uuid: "c885a6d32c3e435ab642c24eb24d91bd"
format_stamp: "Formatted at 2026-06-18 22:18:10 on dist-test-slave-g6r4"
server_key: "c94de0de04736a6a9ce18d8417cf77fd"
server_key_iv: "fee46e7fc8f82929cff783cb98dbcbf3"
server_key_version: "encryptionkey@0"
I20260618 22:18:10.099905 5091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821065380026-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821065380026-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821065380026-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:18:10.104383 5091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:18:10.104954 5091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:18:10.105137 5091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:18:10.105671 5091 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:18:10.106568 5091 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:18:10.106639 5091 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:10.106702 5091 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:18:10.106752 5091 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:10.114974 5091 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:44395
I20260618 22:18:10.115017 5218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:44395 every 8 connection(s)
I20260618 22:18:10.115816 5091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
I20260618 22:18:10.121853 5070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:10.117522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:53859 (local address 127.0.98.126:39187)
0618 22:18:10.117658 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:10.117662 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:10.118669 (+ 1007us) server_negotiation.cc:407] Connection header received
0618 22:18:10.120055 (+ 1386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:10.120059 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:10.120103 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:10.120236 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:10.121384 (+ 1148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:10.121388 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:10.121390 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:18:10.121402 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:10.121424 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:18:10.121433 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:10.121538 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:10.121674 (+ 136us) server_negotiation.cc:299] Negotiation successful
0618 22:18:10.121716 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260618 22:18:10.122627 5221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:10.117882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39187 (local address 127.0.98.65:53859)
0618 22:18:10.118522 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:10.118561 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:10.119812 (+ 1251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:10.120410 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:10.120417 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:10.120500 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:10.121122 (+ 622us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 22:18:10.121124 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:10.121162 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 22:18:10.121225 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:18:10.121524 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:18:10.121532 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:18:10.121537 (+ 5us) client_negotiation.cc:769] Sending connection context
0618 22:18:10.121713 (+ 176us) client_negotiation.cc:240] Negotiation successful
0618 22:18:10.121954 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":100,"threads_started":1}
I20260618 22:18:10.123174 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 5091
I20260618 22:18:10.123265 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821065380026-393-0/minicluster-data/ts-0/wal/instance
I20260618 22:18:10.123478 393 external_mini_cluster.cc:1428] Setting key e367caf42e594040b6cba7ae3de55dd7
I20260618 22:18:10.124543 5219 heartbeater.cc:347] Connected to a master server at 127.0.98.126:39187
I20260618 22:18:10.124755 5219 heartbeater.cc:464] Registering TS with master...
I20260618 22:18:10.125166 5219 heartbeater.cc:511] Master 127.0.98.126:39187 requested a full tablet report, sending...
I20260618 22:18:10.125520 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821065380026-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821065380026-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821065380026-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:39187
--builtin_ntp_servers=127.0.98.84:44323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260618 22:18:10.126642 5030 ts_manager.cc:194] Registered new tserver with Master: c885a6d32c3e435ab642c24eb24d91bd (127.0.98.65:44395)
W20260618 22:18:10.233101 5222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:18:10.233341 5222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:18:10.233378 5222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:18:10.237211 5222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:18:10.237260 5222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:18:10.237311 5222 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260618 22:18:10.241873 5222 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:44323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821065380026-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821065380026-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:39187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5222
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821065380026-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:18:10.242781 5222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:18:10.244029 5222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:18:10.253296 5227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:10.253314 5228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:10.253350 5222 server_base.cc:1034] running on GCE node
W20260618 22:18:10.253376 5230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:10.253835 5222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:18:10.254499 5222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:18:10.255678 5222 hybrid_clock.cc:648] HybridClock initialized: now 1781821090255657 us; error 55 us; skew 500 ppm
I20260618 22:18:10.257351 5222 webserver.cc:460] Webserver started at http://127.0.98.66:38247/ using document root <none> and password file <none>
I20260618 22:18:10.257886 5222 fs_manager.cc:359] Metadata directory not provided
I20260618 22:18:10.258031 5222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:18:10.258247 5222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:18:10.260159 5222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821065380026-393-0/minicluster-data/ts-1/data/instance:
uuid: "ed598eaaae7444af9af17a02300ca737"
format_stamp: "Formatted at 2026-06-18 22:18:10 on dist-test-slave-g6r4"
server_key: "bae515fd89d6c38ead9f23cf63baa394"
server_key_iv: "bebcc2bd1045b17318ef505da30fce80"
server_key_version: "encryptionkey@0"
I20260618 22:18:10.260748 5222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821065380026-393-0/minicluster-data/ts-1/wal/instance:
uuid: "ed598eaaae7444af9af17a02300ca737"
format_stamp: "Formatted at 2026-06-18 22:18:10 on dist-test-slave-g6r4"
server_key: "bae515fd89d6c38ead9f23cf63baa394"
server_key_iv: "bebcc2bd1045b17318ef505da30fce80"
server_key_version: "encryptionkey@0"
I20260618 22:18:10.264302 5222 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260618 22:18:10.266620 5236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:10.267764 5222 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260618 22:18:10.267894 5222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821065380026-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821065380026-393-0/minicluster-data/ts-1/wal
uuid: "ed598eaaae7444af9af17a02300ca737"
format_stamp: "Formatted at 2026-06-18 22:18:10 on dist-test-slave-g6r4"
server_key: "bae515fd89d6c38ead9f23cf63baa394"
server_key_iv: "bebcc2bd1045b17318ef505da30fce80"
server_key_version: "encryptionkey@0"
I20260618 22:18:10.267998 5222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821065380026-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821065380026-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821065380026-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:18:10.272349 5222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:18:10.272912 5222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:18:10.273085 5222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:18:10.273574 5222 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:18:10.274448 5222 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:18:10.274525 5222 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:10.274610 5222 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:18:10.274662 5222 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:10.283072 5222 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:37555
I20260618 22:18:10.283109 5349 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:37555 every 8 connection(s)
I20260618 22:18:10.283890 5222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
I20260618 22:18:10.289749 5070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:10.285656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:46475 (local address 127.0.98.126:39187)
0618 22:18:10.285793 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:10.285797 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:10.286870 (+ 1073us) server_negotiation.cc:407] Connection header received
0618 22:18:10.288142 (+ 1272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:10.288145 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:10.288189 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:10.288272 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:10.289272 (+ 1000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:10.289275 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:10.289278 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:18:10.289289 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:10.289309 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:18:10.289318 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:10.289423 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:10.289561 (+ 138us) server_negotiation.cc:299] Negotiation successful
0618 22:18:10.289608 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260618 22:18:10.290616 5352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:10.286088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39187 (local address 127.0.98.66:46475)
0618 22:18:10.286689 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:10.286758 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:10.287936 (+ 1178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:10.288447 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:10.288453 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:10.288539 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:10.289029 (+ 490us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 22:18:10.289030 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:10.289066 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 22:18:10.289127 (+ 61us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:18:10.289407 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:18:10.289415 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:18:10.289420 (+ 5us) client_negotiation.cc:769] Sending connection context
0618 22:18:10.289654 (+ 234us) client_negotiation.cc:240] Negotiation successful
0618 22:18:10.289888 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":97,"threads_started":1}
I20260618 22:18:10.291373 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 5222
I20260618 22:18:10.291474 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821065380026-393-0/minicluster-data/ts-1/wal/instance
I20260618 22:18:10.291750 393 external_mini_cluster.cc:1428] Setting key 90cf3fd7a3fce9a487b509e5499089be
I20260618 22:18:10.291761 5350 heartbeater.cc:347] Connected to a master server at 127.0.98.126:39187
I20260618 22:18:10.291971 5350 heartbeater.cc:464] Registering TS with master...
I20260618 22:18:10.292392 5350 heartbeater.cc:511] Master 127.0.98.126:39187 requested a full tablet report, sending...
I20260618 22:18:10.293493 5030 ts_manager.cc:194] Registered new tserver with Master: ed598eaaae7444af9af17a02300ca737 (127.0.98.66:37555)
I20260618 22:18:10.293721 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821065380026-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821065380026-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821065380026-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:39187
--builtin_ntp_servers=127.0.98.84:44323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260618 22:18:10.399435 5353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:18:10.399712 5353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:18:10.399782 5353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:18:10.403815 5353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:18:10.403888 5353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:18:10.403980 5353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260618 22:18:10.408591 5353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:44323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821065380026-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821065380026-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:39187
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821065380026-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:18:10.409546 5353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:18:10.410924 5353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:18:10.420212 5361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:10.420236 5358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:10.420377 5353 server_base.cc:1034] running on GCE node
W20260618 22:18:10.420228 5359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:10.421021 5353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:18:10.421972 5353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:18:10.423156 5353 hybrid_clock.cc:648] HybridClock initialized: now 1781821090423134 us; error 33 us; skew 500 ppm
I20260618 22:18:10.425103 5353 webserver.cc:460] Webserver started at http://127.0.98.67:39267/ using document root <none> and password file <none>
I20260618 22:18:10.425673 5353 fs_manager.cc:359] Metadata directory not provided
I20260618 22:18:10.425765 5353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:18:10.426028 5353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:18:10.428006 5353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821065380026-393-0/minicluster-data/ts-2/data/instance:
uuid: "f248c1dc796c4cc2a98f2427aa963f60"
format_stamp: "Formatted at 2026-06-18 22:18:10 on dist-test-slave-g6r4"
server_key: "f2137488e1b9b149405e9e8796c77aa2"
server_key_iv: "5b8001ceaaca342c173a78cfd1c7a006"
server_key_version: "encryptionkey@0"
I20260618 22:18:10.428555 5353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821065380026-393-0/minicluster-data/ts-2/wal/instance:
uuid: "f248c1dc796c4cc2a98f2427aa963f60"
format_stamp: "Formatted at 2026-06-18 22:18:10 on dist-test-slave-g6r4"
server_key: "f2137488e1b9b149405e9e8796c77aa2"
server_key_iv: "5b8001ceaaca342c173a78cfd1c7a006"
server_key_version: "encryptionkey@0"
I20260618 22:18:10.432075 5353 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260618 22:18:10.434413 5367 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:10.435760 5353 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:18:10.435909 5353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821065380026-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821065380026-393-0/minicluster-data/ts-2/wal
uuid: "f248c1dc796c4cc2a98f2427aa963f60"
format_stamp: "Formatted at 2026-06-18 22:18:10 on dist-test-slave-g6r4"
server_key: "f2137488e1b9b149405e9e8796c77aa2"
server_key_iv: "5b8001ceaaca342c173a78cfd1c7a006"
server_key_version: "encryptionkey@0"
I20260618 22:18:10.436025 5353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821065380026-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821065380026-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821065380026-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:18:10.441030 5353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:18:10.441601 5353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:18:10.441778 5353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:18:10.442348 5353 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:18:10.443228 5353 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:18:10.443295 5353 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:10.443359 5353 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:18:10.443408 5353 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:10.452178 5353 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:33779
I20260618 22:18:10.452201 5480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:33779 every 8 connection(s)
I20260618 22:18:10.453066 5353 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
I20260618 22:18:10.458642 5070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:10.454822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:53229 (local address 127.0.98.126:39187)
0618 22:18:10.454947 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:10.454952 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:10.455887 (+ 935us) server_negotiation.cc:407] Connection header received
0618 22:18:10.457110 (+ 1223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:10.457113 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:10.457148 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:10.457221 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:10.458276 (+ 1055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:10.458279 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:10.458281 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:18:10.458294 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:10.458313 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:18:10.458320 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:10.458394 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:10.458509 (+ 115us) server_negotiation.cc:299] Negotiation successful
0618 22:18:10.458545 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260618 22:18:10.459460 5483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:10.455139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39187 (local address 127.0.98.67:53229)
0618 22:18:10.455750 (+ 611us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:10.455789 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:10.456934 (+ 1145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:10.457368 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:10.457374 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:10.457455 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:10.458028 (+ 573us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 22:18:10.458030 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:10.458067 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 22:18:10.458139 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:18:10.458398 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:18:10.458404 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:18:10.458407 (+ 3us) client_negotiation.cc:769] Sending connection context
0618 22:18:10.458525 (+ 118us) client_negotiation.cc:240] Negotiation successful
0618 22:18:10.458701 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":101,"threads_started":1}
I20260618 22:18:10.459498 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 5353
I20260618 22:18:10.459616 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821065380026-393-0/minicluster-data/ts-2/wal/instance
I20260618 22:18:10.459878 393 external_mini_cluster.cc:1428] Setting key d8395ea2cb939b636a74b4adbced5088
I20260618 22:18:10.460667 5481 heartbeater.cc:347] Connected to a master server at 127.0.98.126:39187
I20260618 22:18:10.460902 5481 heartbeater.cc:464] Registering TS with master...
I20260618 22:18:10.461313 5481 heartbeater.cc:511] Master 127.0.98.126:39187 requested a full tablet report, sending...
I20260618 22:18:10.462359 5030 ts_manager.cc:194] Registered new tserver with Master: f248c1dc796c4cc2a98f2427aa963f60 (127.0.98.67:33779)
I20260618 22:18:10.473223 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 22:18:10.476886 5070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:10.475132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49586 (local address 127.0.98.126:39187)
0618 22:18:10.475257 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:10.475261 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:10.475491 (+ 230us) server_negotiation.cc:407] Connection header received
0618 22:18:10.475636 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:10.475639 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:10.475674 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:10.475767 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:10.476489 (+ 722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:10.476493 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:10.476495 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:18:10.476507 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:10.476525 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:18:10.476534 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:10.476633 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:10.476713 (+ 80us) server_negotiation.cc:299] Negotiation successful
0618 22:18:10.476755 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260618 22:18:10.479537 5030 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:49586:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 22:18:10.482460 5030 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:18:10.493436 5493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:10.490414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38422 (local address 127.0.98.65:44395)
0618 22:18:10.491066 (+ 652us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:10.491080 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:10.491424 (+ 344us) server_negotiation.cc:407] Connection header received
0618 22:18:10.491547 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:10.491560 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:10.491711 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:10.491885 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:10.492819 (+ 934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:10.492823 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:10.492831 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:18:10.492839 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:10.492899 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:18:10.492927 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:10.493040 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:10.493186 (+ 146us) server_negotiation.cc:299] Negotiation successful
0618 22:18:10.493261 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":104,"threads_started":1}
I20260618 22:18:10.493988 5494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:10.490728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:37555 (local address 127.0.0.1:42740)
0618 22:18:10.491764 (+ 1036us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:10.491775 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:10.491834 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:10.492966 (+ 1132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:10.492968 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:10.493002 (+ 34us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:10.493366 (+ 364us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 22:18:10.493368 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:10.493377 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 22:18:10.493400 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:18:10.493722 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:18:10.493725 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:18:10.493729 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 22:18:10.493817 (+ 88us) client_negotiation.cc:240] Negotiation successful
0618 22:18:10.493851 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":983,"thread_start_us":87,"threads_started":1}
I20260618 22:18:10.494079 5497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:10.491109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42740 (local address 127.0.98.66:37555)
0618 22:18:10.492048 (+ 939us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:10.492060 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:10.492079 (+ 19us) server_negotiation.cc:407] Connection header received
0618 22:18:10.492138 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:10.492142 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:10.492296 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:10.492422 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:10.493536 (+ 1114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:10.493543 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:10.493556 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:18:10.493569 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:10.493611 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:18:10.493639 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:10.493747 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:10.493833 (+ 86us) server_negotiation.cc:299] Negotiation successful
0618 22:18:10.493894 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":847,"thread_start_us":112,"threads_started":1}
I20260618 22:18:10.494563 5495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:10.490453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:33779 (local address 127.0.0.1:38522)
0618 22:18:10.491471 (+ 1018us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:10.491485 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:10.491593 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:10.492684 (+ 1091us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:10.492690 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:10.492708 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:10.493868 (+ 1160us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 22:18:10.493870 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:10.493879 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 22:18:10.493903 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:18:10.494342 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:18:10.494345 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:18:10.494349 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 22:18:10.494421 (+ 72us) client_negotiation.cc:240] Negotiation successful
0618 22:18:10.494454 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":955,"thread_start_us":83,"threads_started":1}
I20260618 22:18:10.493407 5492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:10.490238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:44395 (local address 127.0.0.1:38422)
0618 22:18:10.491244 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:10.491311 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:10.491446 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:10.491890 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:10.491898 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:10.491972 (+ 74us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:10.492521 (+ 549us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 22:18:10.492525 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:10.492550 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 22:18:10.492619 (+ 69us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:18:10.493019 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:18:10.493028 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:18:10.493035 (+ 7us) client_negotiation.cc:769] Sending connection context
0618 22:18:10.493163 (+ 128us) client_negotiation.cc:240] Negotiation successful
0618 22:18:10.493245 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":77,"threads_started":1}
I20260618 22:18:10.494829 5496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:10.490958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38522 (local address 127.0.98.67:33779)
0618 22:18:10.491276 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:10.491289 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:10.491586 (+ 297us) server_negotiation.cc:407] Connection header received
0618 22:18:10.491696 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:10.491703 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:10.491897 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:10.492023 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:10.494076 (+ 2053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:10.494084 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:10.494094 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:18:10.494108 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:10.494151 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:18:10.494181 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:10.494296 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:10.494468 (+ 172us) server_negotiation.cc:299] Negotiation successful
0618 22:18:10.494534 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":145,"threads_started":1}
I20260618 22:18:10.496507 5284 tablet_service.cc:1467] Processing CreateTablet for tablet 9f1309274cbc4862be1f21481db5cbe3 (DEFAULT_TABLE table=test-table [id=4c788252fb8c4bb2964343360e95b576]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:18:10.496945 5153 tablet_service.cc:1467] Processing CreateTablet for tablet 9f1309274cbc4862be1f21481db5cbe3 (DEFAULT_TABLE table=test-table [id=4c788252fb8c4bb2964343360e95b576]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:18:10.497558 5284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f1309274cbc4862be1f21481db5cbe3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:18:10.497999 5153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f1309274cbc4862be1f21481db5cbe3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:18:10.500417 5415 tablet_service.cc:1467] Processing CreateTablet for tablet 9f1309274cbc4862be1f21481db5cbe3 (DEFAULT_TABLE table=test-table [id=4c788252fb8c4bb2964343360e95b576]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:18:10.501397 5415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f1309274cbc4862be1f21481db5cbe3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:18:10.503794 5498 tablet_bootstrap.cc:492] T 9f1309274cbc4862be1f21481db5cbe3 P c885a6d32c3e435ab642c24eb24d91bd: Bootstrap starting.
I20260618 22:18:10.505398 5499 tablet_bootstrap.cc:492] T 9f1309274cbc4862be1f21481db5cbe3 P ed598eaaae7444af9af17a02300ca737: Bootstrap starting.
I20260618 22:18:10.506083 5498 tablet_bootstrap.cc:654] T 9f1309274cbc4862be1f21481db5cbe3 P c885a6d32c3e435ab642c24eb24d91bd: Neither blocks nor log segments found. Creating new log.
I20260618 22:18:10.506764 5498 log.cc:824] T 9f1309274cbc4862be1f21481db5cbe3 P c885a6d32c3e435ab642c24eb24d91bd: Log is configured to *not* fsync() on all Append() calls
I20260618 22:18:10.507422 5499 tablet_bootstrap.cc:654] T 9f1309274cbc4862be1f21481db5cbe3 P ed598eaaae7444af9af17a02300ca737: Neither blocks nor log segments found. Creating new log.
I20260618 22:18:10.507929 5501 tablet_bootstrap.cc:492] T 9f1309274cbc4862be1f21481db5cbe3 P f248c1dc796c4cc2a98f2427aa963f60: Bootstrap starting.
I20260618 22:18:10.508208 5499 log.cc:824] T 9f1309274cbc4862be1f21481db5cbe3 P ed598eaaae7444af9af17a02300ca737: Log is configured to *not* fsync() on all Append() calls
I20260618 22:18:10.508903 5498 tablet_bootstrap.cc:492] T 9f1309274cbc4862be1f21481db5cbe3 P c885a6d32c3e435ab642c24eb24d91bd: No bootstrap required, opened a new log
I20260618 22:18:10.509080 5498 ts_tablet_manager.cc:1397] T 9f1309274cbc4862be1f21481db5cbe3 P c885a6d32c3e435ab642c24eb24d91bd: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260618 22:18:10.509821 5499 tablet_bootstrap.cc:492] T 9f1309274cbc4862be1f21481db5cbe3 P ed598eaaae7444af9af17a02300ca737: No bootstrap required, opened a new log
I20260618 22:18:10.510017 5501 tablet_bootstrap.cc:654] T 9f1309274cbc4862be1f21481db5cbe3 P f248c1dc796c4cc2a98f2427aa963f60: Neither blocks nor log segments found. Creating new log.
I20260618 22:18:10.510030 5499 ts_tablet_manager.cc:1397] T 9f1309274cbc4862be1f21481db5cbe3 P ed598eaaae7444af9af17a02300ca737: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260618 22:18:10.510775 5501 log.cc:824] T 9f1309274cbc4862be1f21481db5cbe3 P f248c1dc796c4cc2a98f2427aa963f60: Log is configured to *not* fsync() on all Append() calls
I20260618 22:18:10.512938 5498 raft_consensus.cc:348] T 9f1309274cbc4862be1f21481db5cbe3 P c885a6d32c3e435ab642c24eb24d91bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c885a6d32c3e435ab642c24eb24d91bd" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 44395 } } peers { permanent_uuid: "f248c1dc796c4cc2a98f2427aa963f60" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33779 } } peers { permanent_uuid: "ed598eaaae7444af9af17a02300ca737" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37555 } }
I20260618 22:18:10.513137 5498 raft_consensus.cc:374] T 9f1309274cbc4862be1f21481db5cbe3 P c885a6d32c3e435ab642c24eb24d91bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:18:10.513170 5498 raft_consensus.cc:729] T 9f1309274cbc4862be1f21481db5cbe3 P c885a6d32c3e435ab642c24eb24d91bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c885a6d32c3e435ab642c24eb24d91bd, State: Initialized, Role: FOLLOWER
I20260618 22:18:10.513469 5499 raft_consensus.cc:348] T 9f1309274cbc4862be1f21481db5cbe3 P ed598eaaae7444af9af17a02300ca737 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c885a6d32c3e435ab642c24eb24d91bd" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 44395 } } peers { permanent_uuid: "f248c1dc796c4cc2a98f2427aa963f60" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33779 } } peers { permanent_uuid: "ed598eaaae7444af9af17a02300ca737" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37555 } }
I20260618 22:18:10.513679 5499 raft_consensus.cc:374] T 9f1309274cbc4862be1f21481db5cbe3 P ed598eaaae7444af9af17a02300ca737 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:18:10.513612 5498 consensus_queue.cc:260] T 9f1309274cbc4862be1f21481db5cbe3 P c885a6d32c3e435ab642c24eb24d91bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c885a6d32c3e435ab642c24eb24d91bd" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 44395 } } peers { permanent_uuid: "f248c1dc796c4cc2a98f2427aa963f60" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33779 } } peers { permanent_uuid: "ed598eaaae7444af9af17a02300ca737" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37555 } }
I20260618 22:18:10.513736 5499 raft_consensus.cc:729] T 9f1309274cbc4862be1f21481db5cbe3 P ed598eaaae7444af9af17a02300ca737 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed598eaaae7444af9af17a02300ca737, State: Initialized, Role: FOLLOWER
I20260618 22:18:10.514218 5219 heartbeater.cc:503] Master 127.0.98.126:39187 was elected leader, sending a full tablet report...
I20260618 22:18:10.514204 5499 consensus_queue.cc:260] T 9f1309274cbc4862be1f21481db5cbe3 P ed598eaaae7444af9af17a02300ca737 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c885a6d32c3e435ab642c24eb24d91bd" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 44395 } } peers { permanent_uuid: "f248c1dc796c4cc2a98f2427aa963f60" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33779 } } peers { permanent_uuid: "ed598eaaae7444af9af17a02300ca737" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37555 } }
I20260618 22:18:10.514827 5350 heartbeater.cc:503] Master 127.0.98.126:39187 was elected leader, sending a full tablet report...
I20260618 22:18:10.515201 5499 ts_tablet_manager.cc:1428] T 9f1309274cbc4862be1f21481db5cbe3 P ed598eaaae7444af9af17a02300ca737: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260618 22:18:10.515527 5498 ts_tablet_manager.cc:1428] T 9f1309274cbc4862be1f21481db5cbe3 P c885a6d32c3e435ab642c24eb24d91bd: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260618 22:18:10.516301 5501 tablet_bootstrap.cc:492] T 9f1309274cbc4862be1f21481db5cbe3 P f248c1dc796c4cc2a98f2427aa963f60: No bootstrap required, opened a new log
I20260618 22:18:10.516481 5501 ts_tablet_manager.cc:1397] T 9f1309274cbc4862be1f21481db5cbe3 P f248c1dc796c4cc2a98f2427aa963f60: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.003s
I20260618 22:18:10.520289 5501 raft_consensus.cc:348] T 9f1309274cbc4862be1f21481db5cbe3 P f248c1dc796c4cc2a98f2427aa963f60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c885a6d32c3e435ab642c24eb24d91bd" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 44395 } } peers { permanent_uuid: "f248c1dc796c4cc2a98f2427aa963f60" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33779 } } peers { permanent_uuid: "ed598eaaae7444af9af17a02300ca737" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37555 } }
I20260618 22:18:10.520543 5501 raft_consensus.cc:374] T 9f1309274cbc4862be1f21481db5cbe3 P f248c1dc796c4cc2a98f2427aa963f60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:18:10.520622 5501 raft_consensus.cc:729] T 9f1309274cbc4862be1f21481db5cbe3 P f248c1dc796c4cc2a98f2427aa963f60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f248c1dc796c4cc2a98f2427aa963f60, State: Initialized, Role: FOLLOWER
I20260618 22:18:10.521015 5501 consensus_queue.cc:260] T 9f1309274cbc4862be1f21481db5cbe3 P f248c1dc796c4cc2a98f2427aa963f60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c885a6d32c3e435ab642c24eb24d91bd" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 44395 } } peers { permanent_uuid: "f248c1dc796c4cc2a98f2427aa963f60" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33779 } } peers { permanent_uuid: "ed598eaaae7444af9af17a02300ca737" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37555 } }
I20260618 22:18:10.521623 5481 heartbeater.cc:503] Master 127.0.98.126:39187 was elected leader, sending a full tablet report...
I20260618 22:18:10.521826 5501 ts_tablet_manager.cc:1428] T 9f1309274cbc4862be1f21481db5cbe3 P f248c1dc796c4cc2a98f2427aa963f60: Time spent starting tablet: real 0.005s user 0.004s sys 0.002s
W20260618 22:18:10.534772 5351 tablet.cc:2369] T 9f1309274cbc4862be1f21481db5cbe3 P ed598eaaae7444af9af17a02300ca737: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260618 22:18:10.616925 5220 tablet.cc:2369] T 9f1309274cbc4862be1f21481db5cbe3 P c885a6d32c3e435ab642c24eb24d91bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:18:10.639154 5506 raft_consensus.cc:482] T 9f1309274cbc4862be1f21481db5cbe3 P f248c1dc796c4cc2a98f2427aa963f60 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:18:10.639359 5506 raft_consensus.cc:504] T 9f1309274cbc4862be1f21481db5cbe3 P f248c1dc796c4cc2a98f2427aa963f60 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c885a6d32c3e435ab642c24eb24d91bd" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 44395 } } peers { permanent_uuid: "f248c1dc796c4cc2a98f2427aa963f60" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33779 } } peers { permanent_uuid: "ed598eaaae7444af9af17a02300ca737" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37555 } }
I20260618 22:18:10.640553 5506 leader_election.cc:288] T 9f1309274cbc4862be1f21481db5cbe3 P f248c1dc796c4cc2a98f2427aa963f60 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c885a6d32c3e435ab642c24eb24d91bd (127.0.98.65:44395), ed598eaaae7444af9af17a02300ca737 (127.0.98.66:37555)
I20260618 22:18:10.642627 5497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:10.640931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:46741 (local address 127.0.98.66:37555)
0618 22:18:10.641050 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:10.641055 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:10.641353 (+ 298us) server_negotiation.cc:407] Connection header received
0618 22:18:10.641470 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:10.641474 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:10.641516 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:10.641605 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:10.642212 (+ 607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:10.642215 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:10.642217 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:18:10.642229 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:10.642247 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:18:10.642254 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:10.642332 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:10.642460 (+ 128us) server_negotiation.cc:299] Negotiation successful
0618 22:18:10.642501 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260618 22:18:10.642642 5493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:10.640931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:60439 (local address 127.0.98.65:44395)
0618 22:18:10.641050 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:10.641055 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:10.641073 (+ 18us) server_negotiation.cc:407] Connection header received
0618 22:18:10.641127 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:10.641129 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:10.641159 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:10.641261 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:10.642031 (+ 770us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:10.642035 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:10.642037 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:18:10.642050 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:10.642072 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:18:10.642082 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:10.642187 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:10.642484 (+ 297us) server_negotiation.cc:299] Negotiation successful
0618 22:18:10.642524 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260618 22:18:10.642642 5483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:10.640814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:44395 (local address 127.0.98.67:60439)
0618 22:18:10.640919 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:10.640935 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:10.641015 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:10.641285 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:10.641288 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:10.641304 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:10.641823 (+ 519us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 22:18:10.641827 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:10.641839 (+ 12us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 22:18:10.641870 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:18:10.642368 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:18:10.642370 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:18:10.642372 (+ 2us) client_negotiation.cc:769] Sending connection context
0618 22:18:10.642471 (+ 99us) client_negotiation.cc:240] Negotiation successful
0618 22:18:10.642507 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260618 22:18:10.642658 5507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:10.640814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:37555 (local address 127.0.98.67:46741)
0618 22:18:10.641255 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:10.641266 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:10.641371 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:10.641623 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:10.641626 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:10.641639 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:10.642050 (+ 411us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 22:18:10.642052 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:10.642062 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 22:18:10.642086 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:18:10.642332 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:18:10.642335 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:18:10.642337 (+ 2us) client_negotiation.cc:769] Sending connection context
0618 22:18:10.642430 (+ 93us) client_negotiation.cc:240] Negotiation successful
0618 22:18:10.642468 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"mutex_wait_us":125,"thread_start_us":106,"threads_started":1}
I20260618 22:18:10.643321 5173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1309274cbc4862be1f21481db5cbe3" candidate_uuid: "f248c1dc796c4cc2a98f2427aa963f60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c885a6d32c3e435ab642c24eb24d91bd" is_pre_election: true
I20260618 22:18:10.643337 5304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1309274cbc4862be1f21481db5cbe3" candidate_uuid: "f248c1dc796c4cc2a98f2427aa963f60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed598eaaae7444af9af17a02300ca737" is_pre_election: true
I20260618 22:18:10.643602 5304 raft_consensus.cc:2454] T 9f1309274cbc4862be1f21481db5cbe3 P ed598eaaae7444af9af17a02300ca737 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f248c1dc796c4cc2a98f2427aa963f60 in term 0.
I20260618 22:18:10.643600 5173 raft_consensus.cc:2454] T 9f1309274cbc4862be1f21481db5cbe3 P c885a6d32c3e435ab642c24eb24d91bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f248c1dc796c4cc2a98f2427aa963f60 in term 0.
I20260618 22:18:10.644040 5368 leader_election.cc:302] T 9f1309274cbc4862be1f21481db5cbe3 P f248c1dc796c4cc2a98f2427aa963f60 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ed598eaaae7444af9af17a02300ca737, f248c1dc796c4cc2a98f2427aa963f60; no voters:
I20260618 22:18:10.644325 5506 raft_consensus.cc:2781] T 9f1309274cbc4862be1f21481db5cbe3 P f248c1dc796c4cc2a98f2427aa963f60 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:18:10.644403 5506 raft_consensus.cc:482] T 9f1309274cbc4862be1f21481db5cbe3 P f248c1dc796c4cc2a98f2427aa963f60 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:18:10.644445 5506 raft_consensus.cc:3037] T 9f1309274cbc4862be1f21481db5cbe3 P f248c1dc796c4cc2a98f2427aa963f60 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:18:10.645416 5506 raft_consensus.cc:504] T 9f1309274cbc4862be1f21481db5cbe3 P f248c1dc796c4cc2a98f2427aa963f60 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c885a6d32c3e435ab642c24eb24d91bd" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 44395 } } peers { permanent_uuid: "f248c1dc796c4cc2a98f2427aa963f60" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33779 } } peers { permanent_uuid: "ed598eaaae7444af9af17a02300ca737" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37555 } }
I20260618 22:18:10.645713 5506 leader_election.cc:288] T 9f1309274cbc4862be1f21481db5cbe3 P f248c1dc796c4cc2a98f2427aa963f60 [CANDIDATE]: Term 1 election: Requested vote from peers c885a6d32c3e435ab642c24eb24d91bd (127.0.98.65:44395), ed598eaaae7444af9af17a02300ca737 (127.0.98.66:37555)
I20260618 22:18:10.646019 5304 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1309274cbc4862be1f21481db5cbe3" candidate_uuid: "f248c1dc796c4cc2a98f2427aa963f60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed598eaaae7444af9af17a02300ca737"
I20260618 22:18:10.646019 5173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f1309274cbc4862be1f21481db5cbe3" candidate_uuid: "f248c1dc796c4cc2a98f2427aa963f60" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c885a6d32c3e435ab642c24eb24d91bd"
I20260618 22:18:10.646137 5304 raft_consensus.cc:3037] T 9f1309274cbc4862be1f21481db5cbe3 P ed598eaaae7444af9af17a02300ca737 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:18:10.646137 5173 raft_consensus.cc:3037] T 9f1309274cbc4862be1f21481db5cbe3 P c885a6d32c3e435ab642c24eb24d91bd [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:18:10.647348 5304 raft_consensus.cc:2454] T 9f1309274cbc4862be1f21481db5cbe3 P ed598eaaae7444af9af17a02300ca737 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f248c1dc796c4cc2a98f2427aa963f60 in term 1.
I20260618 22:18:10.647348 5173 raft_consensus.cc:2454] T 9f1309274cbc4862be1f21481db5cbe3 P c885a6d32c3e435ab642c24eb24d91bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f248c1dc796c4cc2a98f2427aa963f60 in term 1.
I20260618 22:18:10.647615 5368 leader_election.cc:302] T 9f1309274cbc4862be1f21481db5cbe3 P f248c1dc796c4cc2a98f2427aa963f60 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ed598eaaae7444af9af17a02300ca737, f248c1dc796c4cc2a98f2427aa963f60; no voters:
I20260618 22:18:10.647786 5506 raft_consensus.cc:2781] T 9f1309274cbc4862be1f21481db5cbe3 P f248c1dc796c4cc2a98f2427aa963f60 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:18:10.647986 5506 raft_consensus.cc:686] T 9f1309274cbc4862be1f21481db5cbe3 P f248c1dc796c4cc2a98f2427aa963f60 [term 1 LEADER]: Becoming Leader. State: Replica: f248c1dc796c4cc2a98f2427aa963f60, State: Running, Role: LEADER
I20260618 22:18:10.648290 5506 consensus_queue.cc:237] T 9f1309274cbc4862be1f21481db5cbe3 P f248c1dc796c4cc2a98f2427aa963f60 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c885a6d32c3e435ab642c24eb24d91bd" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 44395 } } peers { permanent_uuid: "f248c1dc796c4cc2a98f2427aa963f60" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33779 } } peers { permanent_uuid: "ed598eaaae7444af9af17a02300ca737" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37555 } }
I20260618 22:18:10.651345 5031 catalog_manager.cc:5495] T 9f1309274cbc4862be1f21481db5cbe3 P f248c1dc796c4cc2a98f2427aa963f60 reported cstate change: term changed from 0 to 1, leader changed from <none> to f248c1dc796c4cc2a98f2427aa963f60 (127.0.98.67). New cstate: current_term: 1 leader_uuid: "f248c1dc796c4cc2a98f2427aa963f60" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c885a6d32c3e435ab642c24eb24d91bd" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 44395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f248c1dc796c4cc2a98f2427aa963f60" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33779 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed598eaaae7444af9af17a02300ca737" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37555 } health_report { overall_health: UNKNOWN } } }
I20260618 22:18:10.680294 5496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:10.678689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38538 (local address 127.0.98.67:33779)
0618 22:18:10.678832 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:10.678836 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:10.678851 (+ 15us) server_negotiation.cc:407] Connection header received
0618 22:18:10.679018 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:10.679020 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:10.679079 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:10.679172 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:10.679882 (+ 710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:10.679886 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:10.679889 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:18:10.679900 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:10.679919 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:18:10.679935 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:10.680024 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:10.680109 (+ 85us) server_negotiation.cc:299] Negotiation successful
0618 22:18:10.680147 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260618 22:18:10.685945 5173 raft_consensus.cc:1261] T 9f1309274cbc4862be1f21481db5cbe3 P c885a6d32c3e435ab642c24eb24d91bd [term 1 FOLLOWER]: Refusing update from remote peer f248c1dc796c4cc2a98f2427aa963f60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:18:10.685948 5304 raft_consensus.cc:1261] T 9f1309274cbc4862be1f21481db5cbe3 P ed598eaaae7444af9af17a02300ca737 [term 1 FOLLOWER]: Refusing update from remote peer f248c1dc796c4cc2a98f2427aa963f60: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:18:10.686641 5506 consensus_queue.cc:1035] T 9f1309274cbc4862be1f21481db5cbe3 P f248c1dc796c4cc2a98f2427aa963f60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed598eaaae7444af9af17a02300ca737" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:18:10.686884 5508 consensus_queue.cc:1035] T 9f1309274cbc4862be1f21481db5cbe3 P f248c1dc796c4cc2a98f2427aa963f60 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c885a6d32c3e435ab642c24eb24d91bd" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 44395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:18:10.694851 5513 mvcc.cc:205] Tried to move back new op lower bound from 7298339187442622464 to 7298339187298037760. Current Snapshot: MvccSnapshot[applied={T|T < 7298339187442622464}]
I20260618 22:18:10.695021 5512 mvcc.cc:205] Tried to move back new op lower bound from 7298339187442622464 to 7298339187298037760. Current Snapshot: MvccSnapshot[applied={T|T < 7298339187442622464}]
I20260618 22:18:10.702100 5031 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:49586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:18:10.702239 5031 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:49586:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:18:10.704563 5031 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1f4580ccda1b4813abdc927a4fce0d8f: Sending DeleteTablet for 3 replicas of tablet 9f1309274cbc4862be1f21481db5cbe3
I20260618 22:18:10.705260 5153 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f1309274cbc4862be1f21481db5cbe3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:18:10 UTC) from {username='slave'} at 127.0.0.1:38422
I20260618 22:18:10.705401 5415 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f1309274cbc4862be1f21481db5cbe3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:18:10 UTC) from {username='slave'} at 127.0.0.1:38522
I20260618 22:18:10.705718 5521 tablet_replica.cc:330] T 9f1309274cbc4862be1f21481db5cbe3 P c885a6d32c3e435ab642c24eb24d91bd: stopping tablet replica
I20260618 22:18:10.705929 5522 tablet_replica.cc:330] T 9f1309274cbc4862be1f21481db5cbe3 P f248c1dc796c4cc2a98f2427aa963f60: stopping tablet replica
I20260618 22:18:10.706135 5521 raft_consensus.cc:2229] T 9f1309274cbc4862be1f21481db5cbe3 P c885a6d32c3e435ab642c24eb24d91bd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:18:10.706198 5522 raft_consensus.cc:2229] T 9f1309274cbc4862be1f21481db5cbe3 P f248c1dc796c4cc2a98f2427aa963f60 [term 1 LEADER]: Raft consensus shutting down.
I20260618 22:18:10.706303 5284 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f1309274cbc4862be1f21481db5cbe3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:18:10 UTC) from {username='slave'} at 127.0.0.1:42740
I20260618 22:18:10.706530 5522 raft_consensus.cc:2258] T 9f1309274cbc4862be1f21481db5cbe3 P f248c1dc796c4cc2a98f2427aa963f60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:18:10.706575 5521 raft_consensus.cc:2258] T 9f1309274cbc4862be1f21481db5cbe3 P c885a6d32c3e435ab642c24eb24d91bd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:18:10.706733 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 5091
I20260618 22:18:10.708560 5522 ts_tablet_manager.cc:1905] T 9f1309274cbc4862be1f21481db5cbe3 P f248c1dc796c4cc2a98f2427aa963f60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:18:10.712082 5522 ts_tablet_manager.cc:1918] T 9f1309274cbc4862be1f21481db5cbe3 P f248c1dc796c4cc2a98f2427aa963f60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:18:10.712181 5522 log.cc:1196] T 9f1309274cbc4862be1f21481db5cbe3 P f248c1dc796c4cc2a98f2427aa963f60: Deleting WAL directory at /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781821065380026-393-0/minicluster-data/ts-2/wal/wals/9f1309274cbc4862be1f21481db5cbe3
I20260618 22:18:10.712424 5523 tablet_replica.cc:330] T 9f1309274cbc4862be1f21481db5cbe3 P ed598eaaae7444af9af17a02300ca737: stopping tablet replica
I20260618 22:18:10.712523 5522 ts_tablet_manager.cc:1939] T 9f1309274cbc4862be1f21481db5cbe3 P f248c1dc796c4cc2a98f2427aa963f60: Deleting consensus metadata
I20260618 22:18:10.712800 5523 raft_consensus.cc:2229] T 9f1309274cbc4862be1f21481db5cbe3 P ed598eaaae7444af9af17a02300ca737 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:18:10.713104 5523 raft_consensus.cc:2258] T 9f1309274cbc4862be1f21481db5cbe3 P ed598eaaae7444af9af17a02300ca737 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:18:10.713804 5017 catalog_manager.cc:4841] TS f248c1dc796c4cc2a98f2427aa963f60 (127.0.98.67:33779): tablet 9f1309274cbc4862be1f21481db5cbe3 (table test-table [id=4c788252fb8c4bb2964343360e95b576]) successfully deleted
I20260618 22:18:10.714455 5523 ts_tablet_manager.cc:1905] T 9f1309274cbc4862be1f21481db5cbe3 P ed598eaaae7444af9af17a02300ca737: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260618 22:18:10.714799 5018 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.65:44395 (error 108)
I20260618 22:18:10.715720 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 5222
I20260618 22:18:10.716109 5492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:10.715755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:44395 (local address 127.0.0.1:38424)
0618 22:18:10.715939 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:10.716014 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.65:44395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260618 22:18:10.716647 5018 catalog_manager.cc:4584] TS c885a6d32c3e435ab642c24eb24d91bd (127.0.98.65:44395): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9f1309274cbc4862be1f21481db5cbe3: Network error: Client connection negotiation failed: client connection to 127.0.98.65:44395: connect: Connection refused (error 111)
I20260618 22:18:10.720711 5492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:10.720461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:37555 (local address 127.0.0.1:42756)
0618 22:18:10.720563 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:10.720640 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.66:37555: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260618 22:18:10.721045 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 5353
I20260618 22:18:10.726698 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 5000
2026-06-18T22:18:10Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1011 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260618 22:18:10.747277 393 mini_oidc.cc:152] Starting JWKS server
I20260618 22:18:10.748715 393 webserver.cc:460] Webserver started at https://127.0.0.1:42069/ using document root <none> and password file <none>
I20260618 22:18:10.748853 393 mini_oidc.cc:189] Starting OIDC Discovery server
I20260618 22:18:10.749063 393 webserver.cc:460] Webserver started at http://127.0.0.1:42909/ using document root <none> and password file <none>
2026-06-18T22:18:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:18:10Z Disabled control of system clock
I20260618 22:18:10.757225 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781821065380026-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781821065380026-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781821065380026-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:39167
--webserver_interface=127.0.98.126
--webserver_port=0
--builtin_ntp_servers=127.0.98.84:40639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:39167
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42069/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781821065380026-393-0/testchainca.cert with env {}
W20260618 22:18:10.862280 5530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:18:10.862550 5530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:18:10.862588 5530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:18:10.866600 5530 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:18:10.866647 5530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:18:10.866662 5530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:18:10.866677 5530 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:18:10.866727 5530 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260618 22:18:10.866757 5530 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260618 22:18:10.866782 5530 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42069/jwks.json
I20260618 22:18:10.871893 5530 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:40639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781821065380026-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781821065380026-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:39167
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:39167
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42069/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781821065380026-393-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781821065380026-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:18:10.872802 5530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:18:10.873713 5530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:18:10.881551 5535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:10.881529 5536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:10.881585 5538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:10.881799 5530 server_base.cc:1034] running on GCE node
I20260618 22:18:10.882416 5530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:18:10.883380 5530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:18:10.884564 5530 hybrid_clock.cc:648] HybridClock initialized: now 1781821090884538 us; error 50 us; skew 500 ppm
I20260618 22:18:10.886322 5530 webserver.cc:460] Webserver started at http://127.0.98.126:33027/ using document root <none> and password file <none>
I20260618 22:18:10.886837 5530 fs_manager.cc:359] Metadata directory not provided
I20260618 22:18:10.886921 5530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:18:10.887125 5530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:18:10.888985 5530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781821065380026-393-0/minicluster-data/master-0/data/instance:
uuid: "b612e3c4b7cd4730be57b606a3605a5b"
format_stamp: "Formatted at 2026-06-18 22:18:10 on dist-test-slave-g6r4"
server_key: "fe906192678694b91951c905bd59febe"
server_key_iv: "a7a2a994cdcef4199d7cdccb072ba55c"
server_key_version: "encryptionkey@0"
I20260618 22:18:10.889544 5530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781821065380026-393-0/minicluster-data/master-0/wal/instance:
uuid: "b612e3c4b7cd4730be57b606a3605a5b"
format_stamp: "Formatted at 2026-06-18 22:18:10 on dist-test-slave-g6r4"
server_key: "fe906192678694b91951c905bd59febe"
server_key_iv: "a7a2a994cdcef4199d7cdccb072ba55c"
server_key_version: "encryptionkey@0"
I20260618 22:18:10.893155 5530 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260618 22:18:10.895398 5544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:10.896540 5530 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260618 22:18:10.896669 5530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781821065380026-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781821065380026-393-0/minicluster-data/master-0/wal
uuid: "b612e3c4b7cd4730be57b606a3605a5b"
format_stamp: "Formatted at 2026-06-18 22:18:10 on dist-test-slave-g6r4"
server_key: "fe906192678694b91951c905bd59febe"
server_key_iv: "a7a2a994cdcef4199d7cdccb072ba55c"
server_key_version: "encryptionkey@0"
I20260618 22:18:10.896772 5530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781821065380026-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781821065380026-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781821065380026-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:18:10.910928 5530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:18:10.911602 5530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:18:10.911772 5530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:18:10.920482 5530 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:39167
I20260618 22:18:10.920503 5596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:39167 every 8 connection(s)
I20260618 22:18:10.921494 5530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
I20260618 22:18:10.921976 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 5530
I20260618 22:18:10.922132 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781821065380026-393-0/minicluster-data/master-0/wal/instance
I20260618 22:18:10.922435 393 external_mini_cluster.cc:1428] Setting key d4ba4bb84dacbe93337be32f9773d494
I20260618 22:18:10.925019 5597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:18:10.931213 5597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b612e3c4b7cd4730be57b606a3605a5b: Bootstrap starting.
I20260618 22:18:10.931710 5600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:10.923705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37842 (local address 127.0.98.126:39167)
0618 22:18:10.924304 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:10.924320 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:10.924354 (+ 34us) server_negotiation.cc:407] Connection header received
0618 22:18:10.925022 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:10.925038 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:10.925479 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:10.925768 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:10.926193 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:10.927138 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:10.928374 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:10.928839 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:10.928964 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:10.929893 (+ 929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:10.929946 (+ 53us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:10.929973 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:18:10.930350 (+ 377us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:10.930417 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:18:10.930438 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:10.930519 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:10.930686 (+ 167us) server_negotiation.cc:299] Negotiation successful
0618 22:18:10.930961 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":137,"threads_started":1}
I20260618 22:18:10.933570 5597 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b612e3c4b7cd4730be57b606a3605a5b: Neither blocks nor log segments found. Creating new log.
I20260618 22:18:10.934470 5597 log.cc:824] T 00000000000000000000000000000000 P b612e3c4b7cd4730be57b606a3605a5b: Log is configured to *not* fsync() on all Append() calls
I20260618 22:18:10.934656 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:18:10.936532 5597 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b612e3c4b7cd4730be57b606a3605a5b: No bootstrap required, opened a new log
I20260618 22:18:10.939841 5597 raft_consensus.cc:348] T 00000000000000000000000000000000 P b612e3c4b7cd4730be57b606a3605a5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b612e3c4b7cd4730be57b606a3605a5b" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39167 } }
I20260618 22:18:10.940037 5597 raft_consensus.cc:374] T 00000000000000000000000000000000 P b612e3c4b7cd4730be57b606a3605a5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:18:10.940104 5597 raft_consensus.cc:729] T 00000000000000000000000000000000 P b612e3c4b7cd4730be57b606a3605a5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b612e3c4b7cd4730be57b606a3605a5b, State: Initialized, Role: FOLLOWER
I20260618 22:18:10.940496 5597 consensus_queue.cc:260] T 00000000000000000000000000000000 P b612e3c4b7cd4730be57b606a3605a5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b612e3c4b7cd4730be57b606a3605a5b" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39167 } }
I20260618 22:18:10.940636 5597 raft_consensus.cc:388] T 00000000000000000000000000000000 P b612e3c4b7cd4730be57b606a3605a5b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:18:10.940709 5597 raft_consensus.cc:482] T 00000000000000000000000000000000 P b612e3c4b7cd4730be57b606a3605a5b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:18:10.940847 5597 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b612e3c4b7cd4730be57b606a3605a5b [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:18:10.941715 5597 raft_consensus.cc:504] T 00000000000000000000000000000000 P b612e3c4b7cd4730be57b606a3605a5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b612e3c4b7cd4730be57b606a3605a5b" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39167 } }
I20260618 22:18:10.942096 5597 leader_election.cc:302] T 00000000000000000000000000000000 P b612e3c4b7cd4730be57b606a3605a5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b612e3c4b7cd4730be57b606a3605a5b; no voters:
I20260618 22:18:10.942418 5597 leader_election.cc:288] T 00000000000000000000000000000000 P b612e3c4b7cd4730be57b606a3605a5b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:18:10.942572 5602 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b612e3c4b7cd4730be57b606a3605a5b [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:18:10.942791 5602 raft_consensus.cc:686] T 00000000000000000000000000000000 P b612e3c4b7cd4730be57b606a3605a5b [term 1 LEADER]: Becoming Leader. State: Replica: b612e3c4b7cd4730be57b606a3605a5b, State: Running, Role: LEADER
I20260618 22:18:10.943074 5602 consensus_queue.cc:237] T 00000000000000000000000000000000 P b612e3c4b7cd4730be57b606a3605a5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b612e3c4b7cd4730be57b606a3605a5b" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39167 } }
I20260618 22:18:10.943522 5597 sys_catalog.cc:564] T 00000000000000000000000000000000 P b612e3c4b7cd4730be57b606a3605a5b [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:18:10.944629 5604 sys_catalog.cc:455] T 00000000000000000000000000000000 P b612e3c4b7cd4730be57b606a3605a5b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b612e3c4b7cd4730be57b606a3605a5b. Latest consensus state: current_term: 1 leader_uuid: "b612e3c4b7cd4730be57b606a3605a5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b612e3c4b7cd4730be57b606a3605a5b" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39167 } } }
I20260618 22:18:10.944758 5604 sys_catalog.cc:458] T 00000000000000000000000000000000 P b612e3c4b7cd4730be57b606a3605a5b [sys.catalog]: This master's current role is: LEADER
I20260618 22:18:10.945035 5603 sys_catalog.cc:455] T 00000000000000000000000000000000 P b612e3c4b7cd4730be57b606a3605a5b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b612e3c4b7cd4730be57b606a3605a5b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b612e3c4b7cd4730be57b606a3605a5b" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39167 } } }
I20260618 22:18:10.945142 5603 sys_catalog.cc:458] T 00000000000000000000000000000000 P b612e3c4b7cd4730be57b606a3605a5b [sys.catalog]: This master's current role is: LEADER
I20260618 22:18:10.945235 5611 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:18:10.947723 5611 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:18:10.953074 5611 catalog_manager.cc:1347] Generated new cluster ID: 7f6f636be3714c4b8c33949b52745a74
I20260618 22:18:10.953133 5611 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:18:10.974926 5611 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:18:10.975822 5611 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:18:10.983192 5611 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b612e3c4b7cd4730be57b606a3605a5b: Generated new TSK 0
I20260618 22:18:10.983872 5611 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260618 22:18:11.010998 5627 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260618 22:18:11.011102 5627 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260618 22:18:11.011570 5600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:10.994964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37852 (local address 127.0.98.126:39167)
0618 22:18:10.995092 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:10.995096 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:10.995332 (+ 236us) server_negotiation.cc:407] Connection header received
0618 22:18:10.995527 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:10.995529 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:10.995571 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:10.995649 (+ 78us) server_negotiation.cc:226] Negotiated authn=JWT
0618 22:18:10.995899 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:10.996409 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:10.997641 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:10.997862 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:10.998013 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:10.998360 (+ 347us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0618 22:18:11.011278 (+ 12918us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42069/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0618 22:18:11.011471 (+ 193us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37852: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42069/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":50}
W20260618 22:18:11.011644 5600 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37852: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42069/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260618 22:18:11.012960 5626 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:18:10.994901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39167 (local address 127.0.0.1:37852)
0618 22:18:10.995235 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:10.995250 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:10.995386 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:10.995683 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:10.995686 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:10.995694 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0618 22:18:10.995809 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:10.995815 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:10.996507 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:10.996511 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:10.997521 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:10.997531 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:10.998043 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:10.998047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:10.998143 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:10.998193 (+ 50us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0618 22:18:11.011476 (+ 13283us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42069/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0618 22:18:11.011716 (+ 240us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.126:39167: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42069/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":122,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.126:39167: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42069/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260618 22:18:11.013700 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 5530
2026-06-18T22:18:11Z chronyd exiting
I20260618 22:18:11.050145 393 test_util.cc:175] -----------------------------------------------
I20260618 22:18:11.050237 393 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781821065380026-393-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (304 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260618 22:18:11.051232 393 mini_oidc.cc:152] Starting JWKS server
I20260618 22:18:11.052095 393 webserver.cc:460] Webserver started at https://127.0.0.1:43761/ using document root <none> and password file <none>
I20260618 22:18:11.052227 393 mini_oidc.cc:189] Starting OIDC Discovery server
I20260618 22:18:11.052523 393 webserver.cc:460] Webserver started at http://127.0.0.1:39901/ using document root <none> and password file <none>
2026-06-18T22:18:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:18:11Z Disabled control of system clock
I20260618 22:18:11.060664 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781821065380026-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781821065380026-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781821065380026-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:36191
--webserver_interface=127.0.98.126
--webserver_port=0
--builtin_ntp_servers=127.0.98.84:43383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:36191
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43761/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781821065380026-393-0/testchainca.cert with env {}
W20260618 22:18:11.166224 5633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:18:11.166450 5633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:18:11.166519 5633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:18:11.170463 5633 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:18:11.170511 5633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:18:11.170526 5633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:18:11.170542 5633 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:18:11.170585 5633 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260618 22:18:11.170617 5633 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260618 22:18:11.170642 5633 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43761/jwks.json
I20260618 22:18:11.175891 5633 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:43383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781821065380026-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781821065380026-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:36191
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:36191
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43761/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781821065380026-393-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5633
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781821065380026-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:18:11.176857 5633 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:18:11.177840 5633 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:18:11.185683 5639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:11.185688 5641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:11.185648 5638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:11.185952 5633 server_base.cc:1034] running on GCE node
I20260618 22:18:11.186441 5633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:18:11.187433 5633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:18:11.188622 5633 hybrid_clock.cc:648] HybridClock initialized: now 1781821091188604 us; error 30 us; skew 500 ppm
I20260618 22:18:11.190474 5633 webserver.cc:460] Webserver started at http://127.0.98.126:39019/ using document root <none> and password file <none>
I20260618 22:18:11.191023 5633 fs_manager.cc:359] Metadata directory not provided
I20260618 22:18:11.191112 5633 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:18:11.191339 5633 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:18:11.193293 5633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781821065380026-393-0/minicluster-data/master-0/data/instance:
uuid: "a502986fc26c4348855b7de798f5b28c"
format_stamp: "Formatted at 2026-06-18 22:18:11 on dist-test-slave-g6r4"
server_key: "83c4519b13a7b5bdddb303f9e914ad91"
server_key_iv: "80ef76fad0b468917859abeb7719a29c"
server_key_version: "encryptionkey@0"
I20260618 22:18:11.193841 5633 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781821065380026-393-0/minicluster-data/master-0/wal/instance:
uuid: "a502986fc26c4348855b7de798f5b28c"
format_stamp: "Formatted at 2026-06-18 22:18:11 on dist-test-slave-g6r4"
server_key: "83c4519b13a7b5bdddb303f9e914ad91"
server_key_iv: "80ef76fad0b468917859abeb7719a29c"
server_key_version: "encryptionkey@0"
I20260618 22:18:11.197270 5633 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260618 22:18:11.199652 5647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:11.200768 5633 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:18:11.200896 5633 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781821065380026-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781821065380026-393-0/minicluster-data/master-0/wal
uuid: "a502986fc26c4348855b7de798f5b28c"
format_stamp: "Formatted at 2026-06-18 22:18:11 on dist-test-slave-g6r4"
server_key: "83c4519b13a7b5bdddb303f9e914ad91"
server_key_iv: "80ef76fad0b468917859abeb7719a29c"
server_key_version: "encryptionkey@0"
I20260618 22:18:11.200995 5633 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781821065380026-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781821065380026-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781821065380026-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:18:11.220913 5633 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:18:11.221545 5633 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:18:11.221736 5633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:18:11.228062 5633 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:36191
I20260618 22:18:11.228106 5699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:36191 every 8 connection(s)
I20260618 22:18:11.229001 5633 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
I20260618 22:18:11.232038 5700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:18:11.235603 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 5633
I20260618 22:18:11.235733 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781821065380026-393-0/minicluster-data/master-0/wal/instance
I20260618 22:18:11.236042 393 external_mini_cluster.cc:1428] Setting key a9ee7bb1398d9f97f79929d3c33e87bb
I20260618 22:18:11.237709 5700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a502986fc26c4348855b7de798f5b28c: Bootstrap starting.
I20260618 22:18:11.239805 5700 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a502986fc26c4348855b7de798f5b28c: Neither blocks nor log segments found. Creating new log.
I20260618 22:18:11.240506 5700 log.cc:824] T 00000000000000000000000000000000 P a502986fc26c4348855b7de798f5b28c: Log is configured to *not* fsync() on all Append() calls
I20260618 22:18:11.242264 5700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a502986fc26c4348855b7de798f5b28c: No bootstrap required, opened a new log
I20260618 22:18:11.244848 5703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:11.237347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43920 (local address 127.0.98.126:36191)
0618 22:18:11.238030 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:11.238046 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:11.238081 (+ 35us) server_negotiation.cc:407] Connection header received
0618 22:18:11.238695 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:11.238710 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:11.239076 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:11.239408 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:11.239811 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:11.240894 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:11.241758 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:11.242278 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:11.242462 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:11.243279 (+ 817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:11.243283 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:11.243302 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:18:11.243653 (+ 351us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:11.243702 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:18:11.243721 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:11.243829 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:11.243972 (+ 143us) server_negotiation.cc:299] Negotiation successful
0618 22:18:11.244202 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":117,"threads_started":1}
I20260618 22:18:11.245873 5700 raft_consensus.cc:348] T 00000000000000000000000000000000 P a502986fc26c4348855b7de798f5b28c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a502986fc26c4348855b7de798f5b28c" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36191 } }
I20260618 22:18:11.246176 5700 raft_consensus.cc:374] T 00000000000000000000000000000000 P a502986fc26c4348855b7de798f5b28c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:18:11.246234 5700 raft_consensus.cc:729] T 00000000000000000000000000000000 P a502986fc26c4348855b7de798f5b28c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a502986fc26c4348855b7de798f5b28c, State: Initialized, Role: FOLLOWER
I20260618 22:18:11.246665 5700 consensus_queue.cc:260] T 00000000000000000000000000000000 P a502986fc26c4348855b7de798f5b28c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a502986fc26c4348855b7de798f5b28c" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36191 } }
I20260618 22:18:11.246815 5700 raft_consensus.cc:388] T 00000000000000000000000000000000 P a502986fc26c4348855b7de798f5b28c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:18:11.246879 5700 raft_consensus.cc:482] T 00000000000000000000000000000000 P a502986fc26c4348855b7de798f5b28c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:18:11.246975 5700 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a502986fc26c4348855b7de798f5b28c [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:18:11.247562 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:18:11.247895 5700 raft_consensus.cc:504] T 00000000000000000000000000000000 P a502986fc26c4348855b7de798f5b28c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a502986fc26c4348855b7de798f5b28c" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36191 } }
I20260618 22:18:11.248203 5700 leader_election.cc:302] T 00000000000000000000000000000000 P a502986fc26c4348855b7de798f5b28c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a502986fc26c4348855b7de798f5b28c; no voters:
I20260618 22:18:11.248519 5700 leader_election.cc:288] T 00000000000000000000000000000000 P a502986fc26c4348855b7de798f5b28c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:18:11.248653 5705 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a502986fc26c4348855b7de798f5b28c [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:18:11.248854 5705 raft_consensus.cc:686] T 00000000000000000000000000000000 P a502986fc26c4348855b7de798f5b28c [term 1 LEADER]: Becoming Leader. State: Replica: a502986fc26c4348855b7de798f5b28c, State: Running, Role: LEADER
I20260618 22:18:11.249230 5705 consensus_queue.cc:237] T 00000000000000000000000000000000 P a502986fc26c4348855b7de798f5b28c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a502986fc26c4348855b7de798f5b28c" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36191 } }
I20260618 22:18:11.249639 5700 sys_catalog.cc:564] T 00000000000000000000000000000000 P a502986fc26c4348855b7de798f5b28c [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:18:11.250414 5706 sys_catalog.cc:455] T 00000000000000000000000000000000 P a502986fc26c4348855b7de798f5b28c [sys.catalog]: SysCatalogTable state changed. Reason: New leader a502986fc26c4348855b7de798f5b28c. Latest consensus state: current_term: 1 leader_uuid: "a502986fc26c4348855b7de798f5b28c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a502986fc26c4348855b7de798f5b28c" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36191 } } }
I20260618 22:18:11.250597 5706 sys_catalog.cc:458] T 00000000000000000000000000000000 P a502986fc26c4348855b7de798f5b28c [sys.catalog]: This master's current role is: LEADER
I20260618 22:18:11.250771 5707 sys_catalog.cc:455] T 00000000000000000000000000000000 P a502986fc26c4348855b7de798f5b28c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a502986fc26c4348855b7de798f5b28c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a502986fc26c4348855b7de798f5b28c" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36191 } } }
I20260618 22:18:11.250878 5707 sys_catalog.cc:458] T 00000000000000000000000000000000 P a502986fc26c4348855b7de798f5b28c [sys.catalog]: This master's current role is: LEADER
W20260618 22:18:11.254410 5721 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a502986fc26c4348855b7de798f5b28c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 22:18:11.254472 5721 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 22:18:11.254549 5714 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:18:11.255226 5714 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:18:11.260428 5714 catalog_manager.cc:1347] Generated new cluster ID: 8b2143c3dfeb434c942f33ae2879304e
I20260618 22:18:11.260486 5714 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:18:11.281561 5714 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:18:11.282644 5714 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:18:11.288687 5714 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a502986fc26c4348855b7de798f5b28c: Generated new TSK 0
I20260618 22:18:11.289428 5714 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260618 22:18:11.319753 5730 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260618 22:18:11.319832 5730 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260618 22:18:11.320382 5703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:11.297627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43928 (local address 127.0.98.126:36191)
0618 22:18:11.297744 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:11.297748 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:11.297888 (+ 140us) server_negotiation.cc:407] Connection header received
0618 22:18:11.298062 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:11.298065 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:11.298108 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:11.298192 (+ 84us) server_negotiation.cc:226] Negotiated authn=JWT
0618 22:18:11.298429 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:11.298931 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:11.300149 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:11.300371 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:11.300446 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:11.300742 (+ 296us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0618 22:18:11.320032 (+ 19290us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43761/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0618 22:18:11.320240 (+ 208us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43928: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43761/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":45}
W20260618 22:18:11.320461 5703 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43928: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43761/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260618 22:18:11.320531 5729 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:18:11.297488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:36191 (local address 127.0.0.1:43928)
0618 22:18:11.297796 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:11.297810 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:11.297945 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:11.298212 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:11.298215 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:11.298223 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0618 22:18:11.298329 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:11.298336 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:11.299036 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:11.299040 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:11.300036 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:11.300046 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:11.300476 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:11.300480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:11.300570 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:11.300597 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0618 22:18:11.320200 (+ 19603us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43761/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0618 22:18:11.320366 (+ 166us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.126:36191: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43761/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":112,"threads_started":1}
I20260618 22:18:11.320920 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 5633
2026-06-18T22:18:11Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (308 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260618 22:18:11.359802 393 mini_oidc.cc:152] Starting JWKS server
I20260618 22:18:11.360679 393 webserver.cc:460] Webserver started at https://127.0.0.1:42463/ using document root <none> and password file <none>
I20260618 22:18:11.360790 393 mini_oidc.cc:189] Starting OIDC Discovery server
I20260618 22:18:11.361092 393 webserver.cc:460] Webserver started at http://127.0.0.1:36165/ using document root <none> and password file <none>
2026-06-18T22:18:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:18:11Z Disabled control of system clock
I20260618 22:18:11.369585 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781821065380026-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781821065380026-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781821065380026-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:44655
--webserver_interface=127.0.98.126
--webserver_port=0
--builtin_ntp_servers=127.0.98.84:33343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:44655
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42463/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260618 22:18:11.475726 5736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:18:11.476018 5736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:18:11.476100 5736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:18:11.480124 5736 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:18:11.480208 5736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:18:11.480237 5736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:18:11.480273 5736 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:18:11.480325 5736 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260618 22:18:11.480378 5736 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260618 22:18:11.480408 5736 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42463/jwks.json
I20260618 22:18:11.485035 5736 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:33343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781821065380026-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781821065380026-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:44655
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:44655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42463/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5736
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781821065380026-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:18:11.486119 5736 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:18:11.487429 5736 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:18:11.495589 5744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:11.495589 5741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:11.495628 5742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:11.495996 5736 server_base.cc:1034] running on GCE node
I20260618 22:18:11.496497 5736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:18:11.497535 5736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:18:11.498692 5736 hybrid_clock.cc:648] HybridClock initialized: now 1781821091498674 us; error 29 us; skew 500 ppm
I20260618 22:18:11.500432 5736 webserver.cc:460] Webserver started at http://127.0.98.126:38761/ using document root <none> and password file <none>
I20260618 22:18:11.500960 5736 fs_manager.cc:359] Metadata directory not provided
I20260618 22:18:11.501019 5736 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:18:11.501175 5736 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:18:11.503144 5736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781821065380026-393-0/minicluster-data/master-0/data/instance:
uuid: "6a6a3e6598044665bd297b076eff2716"
format_stamp: "Formatted at 2026-06-18 22:18:11 on dist-test-slave-g6r4"
server_key: "2e07c940fcc34735e476ef78738c0b7d"
server_key_iv: "dac2c0fa375408e192860bcdfae4c17d"
server_key_version: "encryptionkey@0"
I20260618 22:18:11.503645 5736 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781821065380026-393-0/minicluster-data/master-0/wal/instance:
uuid: "6a6a3e6598044665bd297b076eff2716"
format_stamp: "Formatted at 2026-06-18 22:18:11 on dist-test-slave-g6r4"
server_key: "2e07c940fcc34735e476ef78738c0b7d"
server_key_iv: "dac2c0fa375408e192860bcdfae4c17d"
server_key_version: "encryptionkey@0"
I20260618 22:18:11.507165 5736 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:18:11.509361 5750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:11.510576 5736 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:18:11.510668 5736 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781821065380026-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781821065380026-393-0/minicluster-data/master-0/wal
uuid: "6a6a3e6598044665bd297b076eff2716"
format_stamp: "Formatted at 2026-06-18 22:18:11 on dist-test-slave-g6r4"
server_key: "2e07c940fcc34735e476ef78738c0b7d"
server_key_iv: "dac2c0fa375408e192860bcdfae4c17d"
server_key_version: "encryptionkey@0"
I20260618 22:18:11.510749 5736 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781821065380026-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781821065380026-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781821065380026-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:18:11.525138 5736 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:18:11.525722 5736 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:18:11.525864 5736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:18:11.532425 5736 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:44655
I20260618 22:18:11.532437 5802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:44655 every 8 connection(s)
I20260618 22:18:11.533330 5736 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
I20260618 22:18:11.534576 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 5736
I20260618 22:18:11.534709 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781821065380026-393-0/minicluster-data/master-0/wal/instance
I20260618 22:18:11.534970 393 external_mini_cluster.cc:1428] Setting key 042de36ad6e96d1fce5cc55259a62157
I20260618 22:18:11.536764 5803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:18:11.542973 5803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a6a3e6598044665bd297b076eff2716: Bootstrap starting.
I20260618 22:18:11.543946 5806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:11.536294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48458 (local address 127.0.98.126:44655)
0618 22:18:11.536983 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:11.537000 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:11.537036 (+ 36us) server_negotiation.cc:407] Connection header received
0618 22:18:11.537654 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:11.537669 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:11.538123 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:11.538399 (+ 276us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:11.538726 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:11.539676 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:11.540851 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:11.541281 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:11.541403 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:11.542335 (+ 932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:11.542340 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:11.542351 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:18:11.542716 (+ 365us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:11.542768 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:18:11.542786 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:11.542873 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:11.543030 (+ 157us) server_negotiation.cc:299] Negotiation successful
0618 22:18:11.543279 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":151,"threads_started":1}
I20260618 22:18:11.545260 5803 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a6a3e6598044665bd297b076eff2716: Neither blocks nor log segments found. Creating new log.
I20260618 22:18:11.546205 5803 log.cc:824] T 00000000000000000000000000000000 P 6a6a3e6598044665bd297b076eff2716: Log is configured to *not* fsync() on all Append() calls
I20260618 22:18:11.546830 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:18:11.548415 5803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a6a3e6598044665bd297b076eff2716: No bootstrap required, opened a new log
I20260618 22:18:11.551712 5803 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a6a3e6598044665bd297b076eff2716 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a6a3e6598044665bd297b076eff2716" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 44655 } }
I20260618 22:18:11.551915 5803 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a6a3e6598044665bd297b076eff2716 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:18:11.551982 5803 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a6a3e6598044665bd297b076eff2716 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a6a3e6598044665bd297b076eff2716, State: Initialized, Role: FOLLOWER
I20260618 22:18:11.552382 5803 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a6a3e6598044665bd297b076eff2716 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a6a3e6598044665bd297b076eff2716" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 44655 } }
I20260618 22:18:11.552520 5803 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6a6a3e6598044665bd297b076eff2716 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:18:11.552585 5803 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6a6a3e6598044665bd297b076eff2716 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:18:11.552698 5803 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a6a3e6598044665bd297b076eff2716 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:18:11.553709 5803 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6a6a3e6598044665bd297b076eff2716 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a6a3e6598044665bd297b076eff2716" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 44655 } }
I20260618 22:18:11.554049 5803 leader_election.cc:302] T 00000000000000000000000000000000 P 6a6a3e6598044665bd297b076eff2716 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6a6a3e6598044665bd297b076eff2716; no voters:
I20260618 22:18:11.554359 5803 leader_election.cc:288] T 00000000000000000000000000000000 P 6a6a3e6598044665bd297b076eff2716 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:18:11.554484 5808 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6a6a3e6598044665bd297b076eff2716 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:18:11.554778 5808 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6a6a3e6598044665bd297b076eff2716 [term 1 LEADER]: Becoming Leader. State: Replica: 6a6a3e6598044665bd297b076eff2716, State: Running, Role: LEADER
I20260618 22:18:11.555212 5808 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6a6a3e6598044665bd297b076eff2716 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a6a3e6598044665bd297b076eff2716" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 44655 } }
I20260618 22:18:11.555565 5803 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a6a3e6598044665bd297b076eff2716 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:18:11.556819 5809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a6a3e6598044665bd297b076eff2716 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6a6a3e6598044665bd297b076eff2716" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a6a3e6598044665bd297b076eff2716" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 44655 } } }
I20260618 22:18:11.556932 5809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a6a3e6598044665bd297b076eff2716 [sys.catalog]: This master's current role is: LEADER
I20260618 22:18:11.557047 5810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a6a3e6598044665bd297b076eff2716 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6a6a3e6598044665bd297b076eff2716. Latest consensus state: current_term: 1 leader_uuid: "6a6a3e6598044665bd297b076eff2716" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6a6a3e6598044665bd297b076eff2716" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 44655 } } }
I20260618 22:18:11.557147 5810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a6a3e6598044665bd297b076eff2716 [sys.catalog]: This master's current role is: LEADER
I20260618 22:18:11.557507 5817 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:18:11.560154 5817 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:18:11.565608 5817 catalog_manager.cc:1347] Generated new cluster ID: c95cc6a83cfd4fc59794518f524061c4
I20260618 22:18:11.565676 5817 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:18:11.575964 5817 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:18:11.576790 5817 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:18:11.584882 5817 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6a6a3e6598044665bd297b076eff2716: Generated new TSK 0
I20260618 22:18:11.585469 5817 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:18:11.603318 5806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:11.598594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48466 (local address 127.0.98.126:44655)
0618 22:18:11.598720 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:11.598724 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:11.598932 (+ 208us) server_negotiation.cc:407] Connection header received
0618 22:18:11.599074 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:11.599076 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:11.599118 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:11.599202 (+ 84us) server_negotiation.cc:226] Negotiated authn=JWT
0618 22:18:11.599435 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:11.599945 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:11.601177 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:11.601380 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:11.601453 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:11.601756 (+ 303us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0618 22:18:11.602932 (+ 1176us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42463/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0618 22:18:11.603166 (+ 234us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48466: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42463/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":50}
W20260618 22:18:11.603407 5832 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:18:11.598547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:44655 (local address 127.0.0.1:48466)
0618 22:18:11.598842 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:11.598856 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:11.598965 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:11.599221 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:11.599225 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:11.599232 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0618 22:18:11.599337 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:11.599343 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:11.600050 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:11.600053 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:11.601065 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:11.601075 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:11.601489 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:11.601493 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:11.601581 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:11.601608 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0618 22:18:11.603095 (+ 1487us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42463/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0618 22:18:11.603231 (+ 136us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.126:44655: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42463/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":113,"threads_started":1}
I20260618 22:18:11.603717 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 5736
W20260618 22:18:11.603741 5806 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48466: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42463/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
E20260618 22:18:11.616065 5833 webserver.cc:533] Webserver: sslize failed:
E20260618 22:18:11.616140 5833 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-18T22:18:11Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (284 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-18T22:18:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:18:11Z Disabled control of system clock
I20260618 22:18:11.651964 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781821065380026-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781821065380026-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781821065380026-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:45451
--webserver_interface=127.0.98.126
--webserver_port=0
--builtin_ntp_servers=127.0.98.84:43615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:45451
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781821065380026-393-0/testchainca.cert with env {}
W20260618 22:18:11.761626 5837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:18:11.761919 5837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:18:11.761963 5837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:18:11.765856 5837 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:18:11.765931 5837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:18:11.765949 5837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:18:11.765964 5837 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:18:11.765986 5837 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260618 22:18:11.766008 5837 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260618 22:18:11.766021 5837 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260618 22:18:11.771158 5837 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:43615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781821065380026-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781821065380026-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:45451
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:45451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781821065380026-393-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5837
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781821065380026-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:18:11.772153 5837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:18:11.773080 5837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:18:11.781257 5842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:11.781258 5845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:11.781257 5843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:11.781491 5837 server_base.cc:1034] running on GCE node
I20260618 22:18:11.782172 5837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:18:11.783281 5837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:18:11.784476 5837 hybrid_clock.cc:648] HybridClock initialized: now 1781821091784459 us; error 30 us; skew 500 ppm
I20260618 22:18:11.786391 5837 webserver.cc:460] Webserver started at http://127.0.98.126:45675/ using document root <none> and password file <none>
I20260618 22:18:11.786957 5837 fs_manager.cc:359] Metadata directory not provided
I20260618 22:18:11.787050 5837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:18:11.787277 5837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:18:11.789234 5837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781821065380026-393-0/minicluster-data/master-0/data/instance:
uuid: "b87ab54bc10f494690b402a0eda83d00"
format_stamp: "Formatted at 2026-06-18 22:18:11 on dist-test-slave-g6r4"
server_key: "3a84c0255affc1c69941da29240a81ef"
server_key_iv: "b555b0f31f7f7f6845acff8fb6fced48"
server_key_version: "encryptionkey@0"
I20260618 22:18:11.789785 5837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781821065380026-393-0/minicluster-data/master-0/wal/instance:
uuid: "b87ab54bc10f494690b402a0eda83d00"
format_stamp: "Formatted at 2026-06-18 22:18:11 on dist-test-slave-g6r4"
server_key: "3a84c0255affc1c69941da29240a81ef"
server_key_iv: "b555b0f31f7f7f6845acff8fb6fced48"
server_key_version: "encryptionkey@0"
I20260618 22:18:11.793430 5837 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260618 22:18:11.796176 5851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:11.797350 5837 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:18:11.797483 5837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781821065380026-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781821065380026-393-0/minicluster-data/master-0/wal
uuid: "b87ab54bc10f494690b402a0eda83d00"
format_stamp: "Formatted at 2026-06-18 22:18:11 on dist-test-slave-g6r4"
server_key: "3a84c0255affc1c69941da29240a81ef"
server_key_iv: "b555b0f31f7f7f6845acff8fb6fced48"
server_key_version: "encryptionkey@0"
I20260618 22:18:11.797590 5837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781821065380026-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781821065380026-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781821065380026-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:18:11.820812 5837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:18:11.821481 5837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:18:11.821673 5837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:18:11.828305 5837 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:45451
I20260618 22:18:11.828312 5903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:45451 every 8 connection(s)
I20260618 22:18:11.829190 5837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
I20260618 22:18:11.832101 5904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:18:11.837201 5904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b87ab54bc10f494690b402a0eda83d00: Bootstrap starting.
I20260618 22:18:11.837411 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 5837
I20260618 22:18:11.837550 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781821065380026-393-0/minicluster-data/master-0/wal/instance
I20260618 22:18:11.837808 393 external_mini_cluster.cc:1428] Setting key 10aeea0f70d5ebecb36bf0030e20abc5
I20260618 22:18:11.839804 5904 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b87ab54bc10f494690b402a0eda83d00: Neither blocks nor log segments found. Creating new log.
I20260618 22:18:11.840523 5904 log.cc:824] T 00000000000000000000000000000000 P b87ab54bc10f494690b402a0eda83d00: Log is configured to *not* fsync() on all Append() calls
I20260618 22:18:11.842428 5904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b87ab54bc10f494690b402a0eda83d00: No bootstrap required, opened a new log
I20260618 22:18:11.846587 5904 raft_consensus.cc:348] T 00000000000000000000000000000000 P b87ab54bc10f494690b402a0eda83d00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b87ab54bc10f494690b402a0eda83d00" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 45451 } }
I20260618 22:18:11.846815 5904 raft_consensus.cc:374] T 00000000000000000000000000000000 P b87ab54bc10f494690b402a0eda83d00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:18:11.846895 5904 raft_consensus.cc:729] T 00000000000000000000000000000000 P b87ab54bc10f494690b402a0eda83d00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b87ab54bc10f494690b402a0eda83d00, State: Initialized, Role: FOLLOWER
I20260618 22:18:11.846947 5907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:11.839072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57084 (local address 127.0.98.126:45451)
0618 22:18:11.839641 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:11.839654 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:11.839689 (+ 35us) server_negotiation.cc:407] Connection header received
0618 22:18:11.840343 (+ 654us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:11.840361 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:11.840778 (+ 417us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:11.841144 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:11.841501 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:11.842495 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:11.843691 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:11.844163 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:11.844285 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:11.845087 (+ 802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:11.845092 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:11.845103 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:18:11.845491 (+ 388us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:11.845544 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:18:11.845563 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:11.845635 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:11.845824 (+ 189us) server_negotiation.cc:299] Negotiation successful
0618 22:18:11.846220 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":121,"threads_started":1}
I20260618 22:18:11.847451 5904 consensus_queue.cc:260] T 00000000000000000000000000000000 P b87ab54bc10f494690b402a0eda83d00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b87ab54bc10f494690b402a0eda83d00" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 45451 } }
I20260618 22:18:11.847637 5904 raft_consensus.cc:388] T 00000000000000000000000000000000 P b87ab54bc10f494690b402a0eda83d00 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:18:11.847709 5904 raft_consensus.cc:482] T 00000000000000000000000000000000 P b87ab54bc10f494690b402a0eda83d00 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:18:11.847797 5904 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b87ab54bc10f494690b402a0eda83d00 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:18:11.848680 5904 raft_consensus.cc:504] T 00000000000000000000000000000000 P b87ab54bc10f494690b402a0eda83d00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b87ab54bc10f494690b402a0eda83d00" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 45451 } }
I20260618 22:18:11.849011 5904 leader_election.cc:302] T 00000000000000000000000000000000 P b87ab54bc10f494690b402a0eda83d00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b87ab54bc10f494690b402a0eda83d00; no voters:
I20260618 22:18:11.849349 5904 leader_election.cc:288] T 00000000000000000000000000000000 P b87ab54bc10f494690b402a0eda83d00 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:18:11.849454 5909 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b87ab54bc10f494690b402a0eda83d00 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:18:11.849664 5909 raft_consensus.cc:686] T 00000000000000000000000000000000 P b87ab54bc10f494690b402a0eda83d00 [term 1 LEADER]: Becoming Leader. State: Replica: b87ab54bc10f494690b402a0eda83d00, State: Running, Role: LEADER
I20260618 22:18:11.849805 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:18:11.850054 5909 consensus_queue.cc:237] T 00000000000000000000000000000000 P b87ab54bc10f494690b402a0eda83d00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b87ab54bc10f494690b402a0eda83d00" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 45451 } }
I20260618 22:18:11.850639 5904 sys_catalog.cc:564] T 00000000000000000000000000000000 P b87ab54bc10f494690b402a0eda83d00 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:18:11.851102 5909 sys_catalog.cc:455] T 00000000000000000000000000000000 P b87ab54bc10f494690b402a0eda83d00 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b87ab54bc10f494690b402a0eda83d00. Latest consensus state: current_term: 1 leader_uuid: "b87ab54bc10f494690b402a0eda83d00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b87ab54bc10f494690b402a0eda83d00" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 45451 } } }
I20260618 22:18:11.851219 5909 sys_catalog.cc:458] T 00000000000000000000000000000000 P b87ab54bc10f494690b402a0eda83d00 [sys.catalog]: This master's current role is: LEADER
I20260618 22:18:11.851620 5913 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:18:11.852185 5911 sys_catalog.cc:455] T 00000000000000000000000000000000 P b87ab54bc10f494690b402a0eda83d00 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b87ab54bc10f494690b402a0eda83d00" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b87ab54bc10f494690b402a0eda83d00" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 45451 } } }
I20260618 22:18:11.852284 5911 sys_catalog.cc:458] T 00000000000000000000000000000000 P b87ab54bc10f494690b402a0eda83d00 [sys.catalog]: This master's current role is: LEADER
I20260618 22:18:11.854853 5913 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:18:11.861605 5913 catalog_manager.cc:1347] Generated new cluster ID: 272880bf4e6b4559b87e23f6bd43fba3
I20260618 22:18:11.861696 5913 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:18:11.868721 5913 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:18:11.869728 5913 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:18:11.873848 5913 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b87ab54bc10f494690b402a0eda83d00: Generated new TSK 0
I20260618 22:18:11.874471 5913 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:18:12.339162 5907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:11.881793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57094 (local address 127.0.98.126:45451)
0618 22:18:11.881938 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:11.881942 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:11.882143 (+ 201us) server_negotiation.cc:407] Connection header received
0618 22:18:11.882265 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:11.882268 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:11.882319 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:11.882423 (+ 104us) server_negotiation.cc:226] Negotiated authn=JWT
0618 22:18:11.882663 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:11.883862 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:11.884959 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:11.885309 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:11.885410 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:11.885685 (+ 275us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0618 22:18:12.338772 (+453087us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0618 22:18:12.339043 (+ 271us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57094: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":49}
W20260618 22:18:12.339234 5907 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57094: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260618 22:18:12.339468 5933 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:18:11.881691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:45451 (local address 127.0.0.1:57094)
0618 22:18:11.882062 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:11.882074 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:11.882169 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:11.882459 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:11.882462 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:11.882470 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0618 22:18:11.882559 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:11.882563 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:11.883999 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:11.884002 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:11.884841 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:11.884850 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:11.885438 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:11.885441 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:11.885501 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:11.885523 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0618 22:18:12.339010 (+453487us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0618 22:18:12.339205 (+ 195us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.126:45451: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":93,"threads_started":1}
I20260618 22:18:12.340036 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 5837
2026-06-18T22:18:12Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (714 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (120 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (118 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:18:12 dist-test-slave-g6r4 krb5kdc[5945](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:18:12 dist-test-slave-g6r4 krb5kdc[5945](info): set up 2 sockets
Jun 18 22:18:12 dist-test-slave-g6r4 krb5kdc[5945](info): commencing operation
krb5kdc: starting...
W20260618 22:18:13.622363 393 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/krb5kdc/test-user.keytab.
Jun 18 22:18:13 dist-test-slave-g6r4 krb5kdc[5945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:18:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:18:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260618 22:18:13.738736 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:36121
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:41277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:36121
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:18:13.848462 5958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:18:13.848698 5958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:18:13.848737 5958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:18:13.852735 5958 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:18:13.852792 5958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:18:13.852811 5958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:18:13.852828 5958 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:18:13.852842 5958 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:18:13.857877 5958 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:41277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:36121
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:36121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5958
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:18:13.858847 5958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:18:13.859719 5958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:18:13.867697 5966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:13.867811 5963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:13.867697 5964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:13.868041 5958 server_base.cc:1034] running on GCE node
I20260618 22:18:13.868623 5958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:18:13.869743 5958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:18:13.870926 5958 hybrid_clock.cc:648] HybridClock initialized: now 1781821093870898 us; error 45 us; skew 500 ppm
Jun 18 22:18:13 dist-test-slave-g6r4 krb5kdc[5945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821093, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:18:13.877077 5958 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260618 22:18:13.878137 5958 webserver.cc:460] Webserver started at http://127.0.98.126:42391/ using document root <none> and password file <none>
I20260618 22:18:13.878713 5958 fs_manager.cc:359] Metadata directory not provided
I20260618 22:18:13.878783 5958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:18:13.879014 5958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:18:13.881181 5958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/minicluster-data/master-0/data/instance:
uuid: "22a19224685f4c0987c459ad4f2f4060"
format_stamp: "Formatted at 2026-06-18 22:18:13 on dist-test-slave-g6r4"
server_key: "81e519ad105d23e92e686197b49748cd"
server_key_iv: "7ce9586c51ccc858edf754da944e5476"
server_key_version: "encryptionkey@0"
I20260618 22:18:13.881798 5958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/minicluster-data/master-0/wal/instance:
uuid: "22a19224685f4c0987c459ad4f2f4060"
format_stamp: "Formatted at 2026-06-18 22:18:13 on dist-test-slave-g6r4"
server_key: "81e519ad105d23e92e686197b49748cd"
server_key_iv: "7ce9586c51ccc858edf754da944e5476"
server_key_version: "encryptionkey@0"
I20260618 22:18:13.885607 5958 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:18:13.887962 5973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:13.889146 5958 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:18:13.889292 5958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/minicluster-data/master-0/wal
uuid: "22a19224685f4c0987c459ad4f2f4060"
format_stamp: "Formatted at 2026-06-18 22:18:13 on dist-test-slave-g6r4"
server_key: "81e519ad105d23e92e686197b49748cd"
server_key_iv: "7ce9586c51ccc858edf754da944e5476"
server_key_version: "encryptionkey@0"
I20260618 22:18:13.889398 5958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:18:13.905665 5958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:18:13.908735 5958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:18:13.908931 5958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:18:13.915580 5958 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:36121
I20260618 22:18:13.915859 6025 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:36121 every 8 connection(s)
I20260618 22:18:13.916491 5958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
I20260618 22:18:13.919142 6026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:18:13.924312 6026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22a19224685f4c0987c459ad4f2f4060: Bootstrap starting.
I20260618 22:18:13.924466 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 5958
I20260618 22:18:13.924577 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/minicluster-data/master-0/wal/instance
I20260618 22:18:13.924858 393 external_mini_cluster.cc:1428] Setting key abcf33873a7709c304424bbd9ebd62e7
I20260618 22:18:13.926918 6026 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22a19224685f4c0987c459ad4f2f4060: Neither blocks nor log segments found. Creating new log.
I20260618 22:18:13.927850 6026 log.cc:824] T 00000000000000000000000000000000 P 22a19224685f4c0987c459ad4f2f4060: Log is configured to *not* fsync() on all Append() calls
I20260618 22:18:13.930105 6026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22a19224685f4c0987c459ad4f2f4060: No bootstrap required, opened a new log
Jun 18 22:18:13 dist-test-slave-g6r4 krb5kdc[5945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:18:13.934546 6026 raft_consensus.cc:348] T 00000000000000000000000000000000 P 22a19224685f4c0987c459ad4f2f4060 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22a19224685f4c0987c459ad4f2f4060" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36121 } }
I20260618 22:18:13.934780 6026 raft_consensus.cc:374] T 00000000000000000000000000000000 P 22a19224685f4c0987c459ad4f2f4060 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:18:13.934837 6026 raft_consensus.cc:729] T 00000000000000000000000000000000 P 22a19224685f4c0987c459ad4f2f4060 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22a19224685f4c0987c459ad4f2f4060, State: Initialized, Role: FOLLOWER
I20260618 22:18:13.935220 6026 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22a19224685f4c0987c459ad4f2f4060 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22a19224685f4c0987c459ad4f2f4060" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36121 } }
I20260618 22:18:13.935359 6026 raft_consensus.cc:388] T 00000000000000000000000000000000 P 22a19224685f4c0987c459ad4f2f4060 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:18:13.935423 6026 raft_consensus.cc:482] T 00000000000000000000000000000000 P 22a19224685f4c0987c459ad4f2f4060 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:18:13.935529 6026 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 22a19224685f4c0987c459ad4f2f4060 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:18:13.936314 6026 raft_consensus.cc:504] T 00000000000000000000000000000000 P 22a19224685f4c0987c459ad4f2f4060 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22a19224685f4c0987c459ad4f2f4060" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36121 } }
I20260618 22:18:13.936625 6026 leader_election.cc:302] T 00000000000000000000000000000000 P 22a19224685f4c0987c459ad4f2f4060 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22a19224685f4c0987c459ad4f2f4060; no voters:
I20260618 22:18:13.936926 6026 leader_election.cc:288] T 00000000000000000000000000000000 P 22a19224685f4c0987c459ad4f2f4060 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:18:13.937026 6031 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 22a19224685f4c0987c459ad4f2f4060 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:18:13.937211 6031 raft_consensus.cc:686] T 00000000000000000000000000000000 P 22a19224685f4c0987c459ad4f2f4060 [term 1 LEADER]: Becoming Leader. State: Replica: 22a19224685f4c0987c459ad4f2f4060, State: Running, Role: LEADER
I20260618 22:18:13.937477 6031 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22a19224685f4c0987c459ad4f2f4060 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22a19224685f4c0987c459ad4f2f4060" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36121 } }
I20260618 22:18:13.938112 6026 sys_catalog.cc:564] T 00000000000000000000000000000000 P 22a19224685f4c0987c459ad4f2f4060 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:18:13.939121 6032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22a19224685f4c0987c459ad4f2f4060 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22a19224685f4c0987c459ad4f2f4060" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22a19224685f4c0987c459ad4f2f4060" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36121 } } }
I20260618 22:18:13.939241 6032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22a19224685f4c0987c459ad4f2f4060 [sys.catalog]: This master's current role is: LEADER
I20260618 22:18:13.939550 6033 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22a19224685f4c0987c459ad4f2f4060 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22a19224685f4c0987c459ad4f2f4060. Latest consensus state: current_term: 1 leader_uuid: "22a19224685f4c0987c459ad4f2f4060" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22a19224685f4c0987c459ad4f2f4060" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36121 } } }
I20260618 22:18:13.939644 6033 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22a19224685f4c0987c459ad4f2f4060 [sys.catalog]: This master's current role is: LEADER
I20260618 22:18:13.940222 6040 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:18:13.940563 6029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:13.926142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40656 (local address 127.0.98.126:36121)
0618 22:18:13.926544 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:13.926558 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:13.926590 (+ 32us) server_negotiation.cc:407] Connection header received
0618 22:18:13.927324 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:13.927352 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:13.927785 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:13.928125 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:13.929187 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:13.930507 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:13.931670 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:13.932113 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:13.932222 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:13.935014 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:13.935019 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:13.935053 (+ 34us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:13.935084 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:13.936906 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:13.937409 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:13.937414 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:13.937419 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:13.937551 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:13.937875 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:13.937881 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:13.937883 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:13.938388 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:13.938601 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:13.939305 (+ 704us) server_negotiation.cc:299] Negotiation successful
0618 22:18:13.939623 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":124,"threads_started":1}
I20260618 22:18:13.943048 6040 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:18:13.948159 6040 catalog_manager.cc:1347] Generated new cluster ID: e9ea252185b8425a8d73da58bf133bb1
I20260618 22:18:13.948223 6040 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:18:13.955067 6040 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:18:13.955943 6040 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:18:13.963999 6040 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 22a19224685f4c0987c459ad4f2f4060: Generated new TSK 0
I20260618 22:18:13.964601 6040 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab.
I20260618 22:18:13.993129 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:36121
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:41277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:18:14.114392 6052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:18:14.114624 6052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:18:14.114665 6052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:18:14.118579 6052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:18:14.118626 6052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:18:14.118717 6052 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260618 22:18:14.123694 6052 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:41277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:36121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6052
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:18:14.124598 6052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:18:14.125517 6052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:18:14.134569 6058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:14.134608 6060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:14.134758 6052 server_base.cc:1034] running on GCE node
W20260618 22:18:14.134862 6057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:14.135243 6052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:18:14.135839 6052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:18:14.137028 6052 hybrid_clock.cc:648] HybridClock initialized: now 1781821094136994 us; error 44 us; skew 500 ppm
Jun 18 22:18:14 dist-test-slave-g6r4 krb5kdc[5945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821094, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:18:14.139935 6052 init.cc:377] Logged in from keytab as kudu/127.0.98.65@KRBTEST.COM (short username kudu)
I20260618 22:18:14.140725 6052 webserver.cc:460] Webserver started at http://127.0.98.65:33379/ using document root <none> and password file <none>
I20260618 22:18:14.141248 6052 fs_manager.cc:359] Metadata directory not provided
I20260618 22:18:14.141333 6052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:18:14.141554 6052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:18:14.143461 6052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/minicluster-data/ts-0/data/instance:
uuid: "c032ea78252843d6a8c517510da5531e"
format_stamp: "Formatted at 2026-06-18 22:18:14 on dist-test-slave-g6r4"
server_key: "67e86c22d519444e87882419834a33bc"
server_key_iv: "5f99e1f7abc903ab0a2a4e7dc1005d8b"
server_key_version: "encryptionkey@0"
I20260618 22:18:14.143992 6052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/minicluster-data/ts-0/wal/instance:
uuid: "c032ea78252843d6a8c517510da5531e"
format_stamp: "Formatted at 2026-06-18 22:18:14 on dist-test-slave-g6r4"
server_key: "67e86c22d519444e87882419834a33bc"
server_key_iv: "5f99e1f7abc903ab0a2a4e7dc1005d8b"
server_key_version: "encryptionkey@0"
I20260618 22:18:14.147398 6052 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260618 22:18:14.149566 6067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:14.150724 6052 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:18:14.150851 6052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/minicluster-data/ts-0/wal
uuid: "c032ea78252843d6a8c517510da5531e"
format_stamp: "Formatted at 2026-06-18 22:18:14 on dist-test-slave-g6r4"
server_key: "67e86c22d519444e87882419834a33bc"
server_key_iv: "5f99e1f7abc903ab0a2a4e7dc1005d8b"
server_key_version: "encryptionkey@0"
I20260618 22:18:14.150950 6052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:18:14.166875 6052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:18:14.169952 6052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:18:14.170157 6052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:18:14.170696 6052 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:18:14.171463 6052 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:18:14.171533 6052 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:14.171599 6052 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:18:14.171648 6052 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:14.179895 6052 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:44205
I20260618 22:18:14.179939 6180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:44205 every 8 connection(s)
I20260618 22:18:14.180725 6052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
Jun 18 22:18:14 dist-test-slave-g6r4 krb5kdc[5945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821094, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:18:14.189550 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 6052
I20260618 22:18:14.189692 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/minicluster-data/ts-0/wal/instance
I20260618 22:18:14.190030 393 external_mini_cluster.cc:1428] Setting key 4dc24608ff336e64ada20e33a9601996
I20260618 22:18:14.194366 6029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:14.182349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:56087 (local address 127.0.98.126:36121)
0618 22:18:14.182492 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:14.182496 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:14.183321 (+ 825us) server_negotiation.cc:407] Connection header received
0618 22:18:14.184606 (+ 1285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:14.184608 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:14.184652 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:14.184726 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:14.185817 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:14.186349 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:14.187425 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:14.187651 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:14.187724 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:14.190236 (+ 2512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:14.190240 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:14.190244 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:14.190269 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:14.192023 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:14.192604 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:14.192610 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:14.192614 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:14.192707 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:14.193037 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:14.193043 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:14.193047 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:14.193328 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:14.193407 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:14.194139 (+ 732us) server_negotiation.cc:299] Negotiation successful
0618 22:18:14.194237 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260618 22:18:14.195618 6183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:14.182740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:36121 (local address 127.0.98.65:56087)
0618 22:18:14.183169 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:14.183208 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:14.184419 (+ 1211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:14.184869 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:14.184876 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:14.185205 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:14.185646 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:14.185695 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:14.186451 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:14.186453 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:14.187323 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:14.187332 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:14.187779 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:14.187782 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:14.187914 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:14.188366 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:14.188380 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:14.190043 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:18:14.192171 (+ 2128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:14.192177 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:14.192190 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:14.192512 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:14.192810 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:14.192812 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:14.192813 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:14.192944 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:14.193435 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:18:14.193452 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:18:14.193846 (+ 394us) client_negotiation.cc:769] Sending connection context
0618 22:18:14.194107 (+ 261us) client_negotiation.cc:240] Negotiation successful
0618 22:18:14.194474 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":110,"threads_started":1}
I20260618 22:18:14.196972 6181 heartbeater.cc:347] Connected to a master server at 127.0.98.126:36121
I20260618 22:18:14.197229 6181 heartbeater.cc:464] Registering TS with master...
I20260618 22:18:14.197767 6181 heartbeater.cc:511] Master 127.0.98.126:36121 requested a full tablet report, sending...
I20260618 22:18:14.199429 5990 ts_manager.cc:194] Registered new tserver with Master: c032ea78252843d6a8c517510da5531e (127.0.98.65:44205)
WARNING: no policy specified for kudu/127.0.98.66@KRBTEST.COM; defaulting to no policy
I20260618 22:18:14.200599 5990 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.65@KRBTEST.COM'} at 127.0.98.65:56087
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab.
I20260618 22:18:14.217399 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:36121
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:41277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:18:14.323746 6186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:18:14.324024 6186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:18:14.324098 6186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:18:14.328034 6186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:18:14.328116 6186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:18:14.328202 6186 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260618 22:18:14.333364 6186 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:41277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:36121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6186
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:18:14.334360 6186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:18:14.335296 6186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:18:14.344693 6191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:14.344728 6192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:14.344867 6186 server_base.cc:1034] running on GCE node
W20260618 22:18:14.344767 6194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:14.345254 6186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:18:14.345814 6186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:18:14.346961 6186 hybrid_clock.cc:648] HybridClock initialized: now 1781821094346928 us; error 45 us; skew 500 ppm
Jun 18 22:18:14 dist-test-slave-g6r4 krb5kdc[5945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821094, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:18:14.349506 6186 init.cc:377] Logged in from keytab as kudu/127.0.98.66@KRBTEST.COM (short username kudu)
I20260618 22:18:14.350339 6186 webserver.cc:460] Webserver started at http://127.0.98.66:43587/ using document root <none> and password file <none>
I20260618 22:18:14.350849 6186 fs_manager.cc:359] Metadata directory not provided
I20260618 22:18:14.350935 6186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:18:14.351136 6186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:18:14.353029 6186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/minicluster-data/ts-1/data/instance:
uuid: "366fe9e4beaa495d815cb7278362502d"
format_stamp: "Formatted at 2026-06-18 22:18:14 on dist-test-slave-g6r4"
server_key: "ec2df6492d7b18886bf6ab19e1ed0202"
server_key_iv: "67d3960c3a6c71285806f95ff8aa3b47"
server_key_version: "encryptionkey@0"
I20260618 22:18:14.353572 6186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/minicluster-data/ts-1/wal/instance:
uuid: "366fe9e4beaa495d815cb7278362502d"
format_stamp: "Formatted at 2026-06-18 22:18:14 on dist-test-slave-g6r4"
server_key: "ec2df6492d7b18886bf6ab19e1ed0202"
server_key_iv: "67d3960c3a6c71285806f95ff8aa3b47"
server_key_version: "encryptionkey@0"
I20260618 22:18:14.356992 6186 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260618 22:18:14.359234 6201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:14.360348 6186 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260618 22:18:14.360466 6186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/minicluster-data/ts-1/wal
uuid: "366fe9e4beaa495d815cb7278362502d"
format_stamp: "Formatted at 2026-06-18 22:18:14 on dist-test-slave-g6r4"
server_key: "ec2df6492d7b18886bf6ab19e1ed0202"
server_key_iv: "67d3960c3a6c71285806f95ff8aa3b47"
server_key_version: "encryptionkey@0"
I20260618 22:18:14.360563 6186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:18:14.372200 6186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:18:14.374908 6186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:18:14.375093 6186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:18:14.375617 6186 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:18:14.376367 6186 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:18:14.376433 6186 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:14.376497 6186 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:18:14.376542 6186 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:14.384755 6186 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:38165
I20260618 22:18:14.384790 6314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:38165 every 8 connection(s)
I20260618 22:18:14.385586 6186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
I20260618 22:18:14.393074 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 6186
I20260618 22:18:14.393184 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/minicluster-data/ts-1/wal/instance
I20260618 22:18:14.393450 393 external_mini_cluster.cc:1428] Setting key c607dc63075132a241dc8133cbc72828
Jun 18 22:18:14 dist-test-slave-g6r4 krb5kdc[5945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821094, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:18:14.399701 6029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:14.387326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:54833 (local address 127.0.98.126:36121)
0618 22:18:14.387455 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:14.387458 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:14.388279 (+ 821us) server_negotiation.cc:407] Connection header received
0618 22:18:14.389553 (+ 1274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:14.389555 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:14.389600 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:14.389673 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:14.390940 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:14.391459 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:14.392615 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:14.392872 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:14.392974 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:14.396021 (+ 3047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:14.396024 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:14.396026 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:14.396047 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:14.397527 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:14.398077 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:14.398083 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:14.398085 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:14.398181 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:14.398547 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:14.398550 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:14.398552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:14.398801 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:14.398911 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:14.399415 (+ 504us) server_negotiation.cc:299] Negotiation successful
0618 22:18:14.399536 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260618 22:18:14.400465 6317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:14.387691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:36121 (local address 127.0.98.66:54833)
0618 22:18:14.388110 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:14.388149 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:14.389368 (+ 1219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:14.389978 (+ 610us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:14.389987 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:14.390345 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:14.390809 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:14.390823 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:14.391574 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:14.391577 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:14.392476 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:14.392485 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:14.393023 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:14.393026 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:14.393208 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:14.393869 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:14.393890 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:14.395866 (+ 1976us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:18:14.397644 (+ 1778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:14.397650 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:14.397664 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:14.397978 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:14.398292 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:14.398298 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:14.398301 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:14.398443 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:14.398918 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:18:14.398927 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:18:14.399159 (+ 232us) client_negotiation.cc:769] Sending connection context
0618 22:18:14.399338 (+ 179us) client_negotiation.cc:240] Negotiation successful
0618 22:18:14.399645 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":125,"threads_started":1}
I20260618 22:18:14.401559 6315 heartbeater.cc:347] Connected to a master server at 127.0.98.126:36121
I20260618 22:18:14.401796 6315 heartbeater.cc:464] Registering TS with master...
I20260618 22:18:14.402346 6315 heartbeater.cc:511] Master 127.0.98.126:36121 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.67@KRBTEST.COM; defaulting to no policy
I20260618 22:18:14.403440 5990 ts_manager.cc:194] Registered new tserver with Master: 366fe9e4beaa495d815cb7278362502d (127.0.98.66:38165)
I20260618 22:18:14.404011 5990 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.66@KRBTEST.COM'} at 127.0.98.66:54833
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab.
I20260618 22:18:14.418928 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:36121
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:41277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:18:14.525617 6320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:18:14.525892 6320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:18:14.525982 6320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:18:14.529867 6320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:18:14.529971 6320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:18:14.530098 6320 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260618 22:18:14.535118 6320 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:41277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:36121
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:18:14.536090 6320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:18:14.537050 6320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:18:14.546710 6325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:14.546687 6326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:14.546702 6328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:14.546749 6320 server_base.cc:1034] running on GCE node
I20260618 22:18:14.547251 6320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:18:14.547891 6320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:18:14.549072 6320 hybrid_clock.cc:648] HybridClock initialized: now 1781821094549060 us; error 29 us; skew 500 ppm
Jun 18 22:18:14 dist-test-slave-g6r4 krb5kdc[5945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821094, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:18:14.551767 6320 init.cc:377] Logged in from keytab as kudu/127.0.98.67@KRBTEST.COM (short username kudu)
I20260618 22:18:14.552515 6320 webserver.cc:460] Webserver started at http://127.0.98.67:35061/ using document root <none> and password file <none>
I20260618 22:18:14.553004 6320 fs_manager.cc:359] Metadata directory not provided
I20260618 22:18:14.553058 6320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:18:14.553207 6320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:18:14.555203 6320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/minicluster-data/ts-2/data/instance:
uuid: "ff836028921f46f0876f748d21a9565a"
format_stamp: "Formatted at 2026-06-18 22:18:14 on dist-test-slave-g6r4"
server_key: "3467a1f6fd3842bb7f34b7284ad91d1a"
server_key_iv: "cf04fe39ee089b8c65cdb636a9548b8e"
server_key_version: "encryptionkey@0"
I20260618 22:18:14.555701 6320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/minicluster-data/ts-2/wal/instance:
uuid: "ff836028921f46f0876f748d21a9565a"
format_stamp: "Formatted at 2026-06-18 22:18:14 on dist-test-slave-g6r4"
server_key: "3467a1f6fd3842bb7f34b7284ad91d1a"
server_key_iv: "cf04fe39ee089b8c65cdb636a9548b8e"
server_key_version: "encryptionkey@0"
I20260618 22:18:14.559175 6320 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:18:14.561381 6335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:14.562628 6320 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:18:14.562744 6320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/minicluster-data/ts-2/wal
uuid: "ff836028921f46f0876f748d21a9565a"
format_stamp: "Formatted at 2026-06-18 22:18:14 on dist-test-slave-g6r4"
server_key: "3467a1f6fd3842bb7f34b7284ad91d1a"
server_key_iv: "cf04fe39ee089b8c65cdb636a9548b8e"
server_key_version: "encryptionkey@0"
I20260618 22:18:14.562853 6320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:18:14.581039 6320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:18:14.583869 6320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:18:14.584057 6320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:18:14.584599 6320 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:18:14.585594 6320 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:18:14.585662 6320 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:14.585723 6320 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:18:14.585762 6320 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:14.594333 6320 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:45953
I20260618 22:18:14.594357 6448 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:45953 every 8 connection(s)
I20260618 22:18:14.595145 6320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
Jun 18 22:18:14 dist-test-slave-g6r4 krb5kdc[5945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821094, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:18:14.604916 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 6320
I20260618 22:18:14.605041 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/minicluster-data/ts-2/wal/instance
I20260618 22:18:14.605360 393 external_mini_cluster.cc:1428] Setting key 1e4d8bdcd7126891551e9d0260f33730
I20260618 22:18:14.609164 6029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:14.597182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:44459 (local address 127.0.98.126:36121)
0618 22:18:14.597307 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:14.597312 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:14.598144 (+ 832us) server_negotiation.cc:407] Connection header received
0618 22:18:14.599559 (+ 1415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:14.599563 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:14.599621 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:14.599700 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:14.600933 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:14.601461 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:14.602659 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:14.602892 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:14.603002 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:14.605381 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:14.605385 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:14.605388 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:14.605419 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:14.607150 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:14.607679 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:14.607685 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:14.607687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:14.607773 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:14.608112 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:14.608116 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:14.608117 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:14.608308 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:14.608417 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:14.608892 (+ 475us) server_negotiation.cc:299] Negotiation successful
0618 22:18:14.609002 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260618 22:18:14.609951 6451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:14.597486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:36121 (local address 127.0.98.67:44459)
0618 22:18:14.598005 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:14.598047 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:14.599356 (+ 1309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:14.599925 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:14.599938 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:14.600338 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:14.600789 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:14.600802 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:14.601595 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:14.601599 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:14.602534 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:14.602543 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:14.603050 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:14.603054 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:14.603217 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:14.603777 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:14.603792 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:14.605224 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:18:14.607262 (+ 2038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:14.607268 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:14.607282 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:14.607569 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:14.607868 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:14.607872 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:14.607874 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:14.608007 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:14.608413 (+ 406us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:18:14.608418 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:18:14.608650 (+ 232us) client_negotiation.cc:769] Sending connection context
0618 22:18:14.608849 (+ 199us) client_negotiation.cc:240] Negotiation successful
0618 22:18:14.609146 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":113,"threads_started":1}
I20260618 22:18:14.610966 6449 heartbeater.cc:347] Connected to a master server at 127.0.98.126:36121
I20260618 22:18:14.611215 6449 heartbeater.cc:464] Registering TS with master...
I20260618 22:18:14.611694 6449 heartbeater.cc:511] Master 127.0.98.126:36121 requested a full tablet report, sending...
I20260618 22:18:14.612655 5990 ts_manager.cc:194] Registered new tserver with Master: ff836028921f46f0876f748d21a9565a (127.0.98.67:45953)
I20260618 22:18:14.613193 5990 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.67@KRBTEST.COM'} at 127.0.98.67:44459
I20260618 22:18:14.619313 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/krb5kdc/test-admin.keytab.
Jun 18 22:18:14 dist-test-slave-g6r4 krb5kdc[5945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:18:14.634398 393 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260618 22:18:14.637176 6456 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:18:14.635779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45562 (local address 127.0.98.65:44205)
0618 22:18:14.636118 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:14.636134 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:14.636156 (+ 22us) server_negotiation.cc:407] Connection header received
0618 22:18:14.636204 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:14.636210 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:14.636431 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:14.636579 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:14.636973 (+ 394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45562: BlockingRecv error: recv got EOF from 127.0.0.1:45562 (error 108)
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":153,"threads_started":1}
W20260618 22:18:14.637468 393 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/krb5kdc/krb5cc' of type FILE
Jun 18 22:18:14 dist-test-slave-g6r4 krb5kdc[5945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:18:14.638639 393 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 18 22:18:14 dist-test-slave-g6r4 krb5kdc[5945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:18:14.639613 393 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 18 22:18:14 dist-test-slave-g6r4 krb5kdc[5945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.65@KRBTEST.COM
I20260618 22:18:14.649775 6456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:14.640542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45570 (local address 127.0.98.65:44205)
0618 22:18:14.640652 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:14.640656 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:14.640802 (+ 146us) server_negotiation.cc:407] Connection header received
0618 22:18:14.640949 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:14.640952 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:14.640997 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:14.641081 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:14.641629 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:14.642499 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:14.643379 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:14.643794 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:14.644007 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:14.646351 (+ 2344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:14.646359 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:14.646372 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:14.646403 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:14.648032 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:14.648476 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:14.648484 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:14.648490 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:14.648565 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:14.648835 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:14.648840 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:14.648842 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:14.649079 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:14.649310 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:14.649476 (+ 166us) server_negotiation.cc:299] Negotiation successful
0618 22:18:14.649623 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260618 22:18:14.653501 6456 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:18:14.652494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45580 (local address 127.0.98.65:44205)
0618 22:18:14.652617 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:14.652621 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:14.652728 (+ 107us) server_negotiation.cc:407] Connection header received
0618 22:18:14.652870 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:14.652873 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:14.652916 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:14.652995 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:14.653384 (+ 389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45580: BlockingRecv error: recv got EOF from 127.0.0.1:45580 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260618 22:18:14.653631 393 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/krb5kdc/krb5cc' of type FILE
Jun 18 22:18:14 dist-test-slave-g6r4 krb5kdc[5945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:18:14.654659 393 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 18 22:18:14 dist-test-slave-g6r4 krb5kdc[5945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:18:14.655572 393 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 18 22:18:14 dist-test-slave-g6r4 krb5kdc[5945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.65@KRBTEST.COM
I20260618 22:18:14.663704 6456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:14.656321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45590 (local address 127.0.98.65:44205)
0618 22:18:14.656430 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:14.656435 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:14.656547 (+ 112us) server_negotiation.cc:407] Connection header received
0618 22:18:14.656623 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:14.656625 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:14.656655 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:14.656717 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:14.657154 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:14.657635 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:14.658459 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:14.658674 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:14.658750 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:14.661088 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:14.661091 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:14.661093 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:14.661113 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:14.662359 (+ 1246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:14.662735 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:14.662737 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:14.662738 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:14.662810 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:14.663031 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:14.663033 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:14.663034 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:14.663169 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:14.663259 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:14.663446 (+ 187us) server_negotiation.cc:299] Negotiation successful
0618 22:18:14.663527 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260618 22:18:14.666292 6456 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:18:14.665349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45592 (local address 127.0.98.65:44205)
0618 22:18:14.665475 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:14.665485 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:14.665555 (+ 70us) server_negotiation.cc:407] Connection header received
0618 22:18:14.665661 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:14.665663 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:14.665693 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:14.665789 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:14.666212 (+ 423us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45592: BlockingRecv error: recv got EOF from 127.0.0.1:45592 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260618 22:18:14.666435 393 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781821065380026-393-0/krb5kdc/krb5cc' of type FILE
Jun 18 22:18:14 dist-test-slave-g6r4 krb5kdc[5945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:18:14.667527 393 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 18 22:18:14 dist-test-slave-g6r4 krb5kdc[5945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:18:14.668416 393 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 18 22:18:14 dist-test-slave-g6r4 krb5kdc[5945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821094, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.65@KRBTEST.COM
I20260618 22:18:14.677589 6456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:14.669145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45608 (local address 127.0.98.65:44205)
0618 22:18:14.669290 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:14.669294 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:14.669412 (+ 118us) server_negotiation.cc:407] Connection header received
0618 22:18:14.669553 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:14.669556 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:14.669593 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:14.669699 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:14.670265 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:14.670793 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:14.671652 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:14.671900 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:14.672015 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:14.674337 (+ 2322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:14.674341 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:14.674343 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:14.674367 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:14.675921 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:14.676363 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:14.676369 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:14.676371 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:14.676451 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:14.676744 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:14.676748 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:14.676750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:14.676904 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:14.677058 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:14.677342 (+ 284us) server_negotiation.cc:299] Negotiation successful
0618 22:18:14.677455 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260618 22:18:14.678686 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 6052
I20260618 22:18:14.684273 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 6186
I20260618 22:18:14.690162 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 6320
I20260618 22:18:14.695645 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 5958
2026-06-18T22:18:14Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2118 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:18:14 dist-test-slave-g6r4 krb5kdc[6469](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:18:14 dist-test-slave-g6r4 krb5kdc[6469](info): set up 2 sockets
Jun 18 22:18:14 dist-test-slave-g6r4 krb5kdc[6469](info): commencing operation
krb5kdc: starting...
W20260618 22:18:15.754312 393 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/krb5kdc/test-user.keytab.
Jun 18 22:18:15 dist-test-slave-g6r4 krb5kdc[6469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:18:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:18:15Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/krb5kdc/oryx_127.0.98.126.keytab.
Entry for principal oryx/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/krb5kdc/oryx_127.0.98.126.keytab.
I20260618 22:18:15.870239 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:42267
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/krb5kdc/oryx_127.0.98.126.keytab
--principal=oryx/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:34497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:42267
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:18:15.978076 6482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:18:15.978313 6482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:18:15.978353 6482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:18:15.982312 6482 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:18:15.982362 6482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:18:15.982380 6482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:18:15.982396 6482 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:18:15.982412 6482 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:18:15.987558 6482 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:34497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:42267
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/krb5kdc/oryx_127.0.98.126.keytab
--principal=oryx/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:42267
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6482
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:18:15.988523 6482 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:18:15.989490 6482 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:18:15.997648 6488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:15.997653 6487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:15.997710 6490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:15.997958 6482 server_base.cc:1034] running on GCE node
I20260618 22:18:15.998548 6482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:18:15.999533 6482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:18:16.000699 6482 hybrid_clock.cc:648] HybridClock initialized: now 1781821096000685 us; error 25 us; skew 500 ppm
Jun 18 22:18:15 dist-test-slave-g6r4 krb5kdc[6469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821095, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:18:16.003449 6482 init.cc:377] Logged in from keytab as oryx/127.0.98.126@KRBTEST.COM (short username oryx)
I20260618 22:18:16.004190 6482 webserver.cc:460] Webserver started at http://127.0.98.126:34533/ using document root <none> and password file <none>
I20260618 22:18:16.004666 6482 fs_manager.cc:359] Metadata directory not provided
I20260618 22:18:16.004721 6482 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:18:16.004915 6482 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:18:16.006888 6482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/master-0/data/instance:
uuid: "cafe178b94854b17acaae5da06c1920d"
format_stamp: "Formatted at 2026-06-18 22:18:15 on dist-test-slave-g6r4"
server_key: "2e9f7cb665dbb9736b4ba67181785d4e"
server_key_iv: "21e111d80e0442f9a44a1b447619b62a"
server_key_version: "encryptionkey@0"
I20260618 22:18:16.007424 6482 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/master-0/wal/instance:
uuid: "cafe178b94854b17acaae5da06c1920d"
format_stamp: "Formatted at 2026-06-18 22:18:15 on dist-test-slave-g6r4"
server_key: "2e9f7cb665dbb9736b4ba67181785d4e"
server_key_iv: "21e111d80e0442f9a44a1b447619b62a"
server_key_version: "encryptionkey@0"
I20260618 22:18:16.010941 6482 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:18:16.013289 6497 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:16.014566 6482 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:18:16.014737 6482 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/master-0/wal
uuid: "cafe178b94854b17acaae5da06c1920d"
format_stamp: "Formatted at 2026-06-18 22:18:15 on dist-test-slave-g6r4"
server_key: "2e9f7cb665dbb9736b4ba67181785d4e"
server_key_iv: "21e111d80e0442f9a44a1b447619b62a"
server_key_version: "encryptionkey@0"
I20260618 22:18:16.014846 6482 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:18:16.037084 6482 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:18:16.040182 6482 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:18:16.040382 6482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:18:16.047791 6482 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:42267
I20260618 22:18:16.048072 6549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:42267 every 8 connection(s)
I20260618 22:18:16.048687 6482 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
I20260618 22:18:16.051225 6550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:18:16.055639 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 6482
I20260618 22:18:16.055749 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/master-0/wal/instance
I20260618 22:18:16.056051 393 external_mini_cluster.cc:1428] Setting key 04b5569c4ff1935941618c5bab527764
I20260618 22:18:16.056324 6550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cafe178b94854b17acaae5da06c1920d: Bootstrap starting.
I20260618 22:18:16.058647 6550 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cafe178b94854b17acaae5da06c1920d: Neither blocks nor log segments found. Creating new log.
I20260618 22:18:16.059525 6550 log.cc:824] T 00000000000000000000000000000000 P cafe178b94854b17acaae5da06c1920d: Log is configured to *not* fsync() on all Append() calls
I20260618 22:18:16.061448 6550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cafe178b94854b17acaae5da06c1920d: No bootstrap required, opened a new log
Jun 18 22:18:16 dist-test-slave-g6r4 krb5kdc[6469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM
I20260618 22:18:16.065665 6550 raft_consensus.cc:348] T 00000000000000000000000000000000 P cafe178b94854b17acaae5da06c1920d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cafe178b94854b17acaae5da06c1920d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42267 } }
I20260618 22:18:16.065886 6550 raft_consensus.cc:374] T 00000000000000000000000000000000 P cafe178b94854b17acaae5da06c1920d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:18:16.066008 6550 raft_consensus.cc:729] T 00000000000000000000000000000000 P cafe178b94854b17acaae5da06c1920d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cafe178b94854b17acaae5da06c1920d, State: Initialized, Role: FOLLOWER
I20260618 22:18:16.066520 6550 consensus_queue.cc:260] T 00000000000000000000000000000000 P cafe178b94854b17acaae5da06c1920d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cafe178b94854b17acaae5da06c1920d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42267 } }
I20260618 22:18:16.066685 6550 raft_consensus.cc:388] T 00000000000000000000000000000000 P cafe178b94854b17acaae5da06c1920d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:18:16.066758 6550 raft_consensus.cc:482] T 00000000000000000000000000000000 P cafe178b94854b17acaae5da06c1920d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:18:16.066857 6550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cafe178b94854b17acaae5da06c1920d [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:18:16.067909 6550 raft_consensus.cc:504] T 00000000000000000000000000000000 P cafe178b94854b17acaae5da06c1920d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cafe178b94854b17acaae5da06c1920d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42267 } }
I20260618 22:18:16.068257 6550 leader_election.cc:302] T 00000000000000000000000000000000 P cafe178b94854b17acaae5da06c1920d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cafe178b94854b17acaae5da06c1920d; no voters:
I20260618 22:18:16.068589 6550 leader_election.cc:288] T 00000000000000000000000000000000 P cafe178b94854b17acaae5da06c1920d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:18:16.068684 6555 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cafe178b94854b17acaae5da06c1920d [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:18:16.068888 6555 raft_consensus.cc:686] T 00000000000000000000000000000000 P cafe178b94854b17acaae5da06c1920d [term 1 LEADER]: Becoming Leader. State: Replica: cafe178b94854b17acaae5da06c1920d, State: Running, Role: LEADER
I20260618 22:18:16.069214 6555 consensus_queue.cc:237] T 00000000000000000000000000000000 P cafe178b94854b17acaae5da06c1920d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cafe178b94854b17acaae5da06c1920d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42267 } }
I20260618 22:18:16.069864 6550 sys_catalog.cc:564] T 00000000000000000000000000000000 P cafe178b94854b17acaae5da06c1920d [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:18:16.071081 6553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:16.057297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59692 (local address 127.0.98.126:42267)
0618 22:18:16.057699 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:16.057719 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:16.057753 (+ 34us) server_negotiation.cc:407] Connection header received
0618 22:18:16.058392 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:16.058408 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:16.058721 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:16.059036 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:16.059956 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:16.061040 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:16.062175 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:16.062619 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:16.062744 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:16.065484 (+ 2740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:16.065488 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:16.065500 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:16.065524 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:16.067517 (+ 1993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:16.067954 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:16.067959 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:16.067965 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:16.068091 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:16.068342 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:16.068346 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:16.068348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:16.068858 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:16.069658 (+ 800us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:16.069810 (+ 152us) server_negotiation.cc:299] Negotiation successful
0618 22:18:16.070121 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":129,"threads_started":1}
I20260618 22:18:16.071148 6557 sys_catalog.cc:455] T 00000000000000000000000000000000 P cafe178b94854b17acaae5da06c1920d [sys.catalog]: SysCatalogTable state changed. Reason: New leader cafe178b94854b17acaae5da06c1920d. Latest consensus state: current_term: 1 leader_uuid: "cafe178b94854b17acaae5da06c1920d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cafe178b94854b17acaae5da06c1920d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42267 } } }
I20260618 22:18:16.071265 6557 sys_catalog.cc:458] T 00000000000000000000000000000000 P cafe178b94854b17acaae5da06c1920d [sys.catalog]: This master's current role is: LEADER
I20260618 22:18:16.071385 6556 sys_catalog.cc:455] T 00000000000000000000000000000000 P cafe178b94854b17acaae5da06c1920d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cafe178b94854b17acaae5da06c1920d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cafe178b94854b17acaae5da06c1920d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42267 } } }
I20260618 22:18:16.071462 6556 sys_catalog.cc:458] T 00000000000000000000000000000000 P cafe178b94854b17acaae5da06c1920d [sys.catalog]: This master's current role is: LEADER
I20260618 22:18:16.071923 6564 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:18:16.075168 6564 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:18:16.080819 6564 catalog_manager.cc:1347] Generated new cluster ID: fc2c6ba654e446178226c06313f3379d
I20260618 22:18:16.080914 6564 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:18:16.096747 6564 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:18:16.098400 6564 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:18:16.103477 6564 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cafe178b94854b17acaae5da06c1920d: Generated new TSK 0
I20260618 22:18:16.104178 6564 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/krb5kdc/oryx_127.0.98.65.keytab.
Entry for principal oryx/127.0.98.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/krb5kdc/oryx_127.0.98.65.keytab.
I20260618 22:18:16.130558 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:42267
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/krb5kdc/oryx_127.0.98.65.keytab
--principal=oryx/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:34497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:18:16.237914 6576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:18:16.238150 6576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:18:16.238190 6576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:18:16.242175 6576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:18:16.242226 6576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:18:16.242316 6576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260618 22:18:16.247337 6576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:34497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/krb5kdc/oryx_127.0.98.65.keytab
--principal=oryx/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:42267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:18:16.248303 6576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:18:16.249255 6576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:18:16.258250 6584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:16.258253 6581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:16.258236 6582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:16.258494 6576 server_base.cc:1034] running on GCE node
I20260618 22:18:16.258878 6576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:18:16.259488 6576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:18:16.260670 6576 hybrid_clock.cc:648] HybridClock initialized: now 1781821096260650 us; error 33 us; skew 500 ppm
Jun 18 22:18:16 dist-test-slave-g6r4 krb5kdc[6469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821096, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:18:16.263415 6576 init.cc:377] Logged in from keytab as oryx/127.0.98.65@KRBTEST.COM (short username oryx)
I20260618 22:18:16.264132 6576 webserver.cc:460] Webserver started at http://127.0.98.65:36469/ using document root <none> and password file <none>
I20260618 22:18:16.264593 6576 fs_manager.cc:359] Metadata directory not provided
I20260618 22:18:16.264649 6576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:18:16.264804 6576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:18:16.266716 6576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-0/data/instance:
uuid: "4c35cb6b49cd461b83b7ea8f6e4dcb54"
format_stamp: "Formatted at 2026-06-18 22:18:16 on dist-test-slave-g6r4"
server_key: "83fb473682df8aef856c7fd6c526c6a4"
server_key_iv: "0f86e99ac01b87aed6ff72fe2ffcd7c8"
server_key_version: "encryptionkey@0"
I20260618 22:18:16.267206 6576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-0/wal/instance:
uuid: "4c35cb6b49cd461b83b7ea8f6e4dcb54"
format_stamp: "Formatted at 2026-06-18 22:18:16 on dist-test-slave-g6r4"
server_key: "83fb473682df8aef856c7fd6c526c6a4"
server_key_iv: "0f86e99ac01b87aed6ff72fe2ffcd7c8"
server_key_version: "encryptionkey@0"
I20260618 22:18:16.270555 6576 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:18:16.272922 6591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:16.274102 6576 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:18:16.274207 6576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-0/wal
uuid: "4c35cb6b49cd461b83b7ea8f6e4dcb54"
format_stamp: "Formatted at 2026-06-18 22:18:16 on dist-test-slave-g6r4"
server_key: "83fb473682df8aef856c7fd6c526c6a4"
server_key_iv: "0f86e99ac01b87aed6ff72fe2ffcd7c8"
server_key_version: "encryptionkey@0"
I20260618 22:18:16.274299 6576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:18:16.286947 6576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:18:16.290359 6576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:18:16.290557 6576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:18:16.291139 6576 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:18:16.291906 6576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:18:16.291952 6576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:16.292013 6576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:18:16.292044 6576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:16.300268 6576 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:34925
I20260618 22:18:16.300290 6704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:34925 every 8 connection(s)
I20260618 22:18:16.301147 6576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
I20260618 22:18:16.306264 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 6576
I20260618 22:18:16.306389 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-0/wal/instance
I20260618 22:18:16.306686 393 external_mini_cluster.cc:1428] Setting key a9d16d1ca8f5a0c5af4655fcef0cec8e
Jun 18 22:18:16 dist-test-slave-g6r4 krb5kdc[6469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821096, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM
I20260618 22:18:16.315148 6553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:16.302871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:33719 (local address 127.0.98.126:42267)
0618 22:18:16.303000 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:16.303003 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:16.303744 (+ 741us) server_negotiation.cc:407] Connection header received
0618 22:18:16.305038 (+ 1294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:16.305042 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:16.305097 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:16.305204 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:16.306372 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:16.307018 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:16.308282 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:16.308515 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:16.308597 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:16.311187 (+ 2590us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:16.311194 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:16.311200 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:16.311231 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:16.312977 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:16.313450 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:16.313457 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:16.313461 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:16.313556 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:16.313882 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:16.313885 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:16.313887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:16.314129 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:16.314248 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:16.314831 (+ 583us) server_negotiation.cc:299] Negotiation successful
0618 22:18:16.314952 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260618 22:18:16.315825 6707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:16.303164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:42267 (local address 127.0.98.65:33719)
0618 22:18:16.303582 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:16.303638 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:16.304836 (+ 1198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:16.305388 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:16.305395 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:16.305721 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:16.306230 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:16.306243 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:16.307166 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:16.307169 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:16.308162 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:16.308173 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:16.308674 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:16.308681 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:16.308917 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:16.309520 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:16.309548 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:16.311030 (+ 1482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:18:16.313089 (+ 2059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:16.313093 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:16.313104 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:16.313348 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:16.313671 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:16.313673 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:16.313674 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:16.313791 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:16.314254 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:18:16.314260 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:18:16.314517 (+ 257us) client_negotiation.cc:769] Sending connection context
0618 22:18:16.314741 (+ 224us) client_negotiation.cc:240] Negotiation successful
0618 22:18:16.315019 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":98,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.66@KRBTEST.COM; defaulting to no policy
I20260618 22:18:16.317008 6705 heartbeater.cc:347] Connected to a master server at 127.0.98.126:42267
I20260618 22:18:16.317257 6705 heartbeater.cc:464] Registering TS with master...
I20260618 22:18:16.317806 6705 heartbeater.cc:511] Master 127.0.98.126:42267 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.66@KRBTEST.COM" created.
I20260618 22:18:16.319336 6514 ts_manager.cc:194] Registered new tserver with Master: 4c35cb6b49cd461b83b7ea8f6e4dcb54 (127.0.98.65:34925)
I20260618 22:18:16.320395 6514 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.65@KRBTEST.COM'} at 127.0.98.65:33719
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/krb5kdc/oryx_127.0.98.66.keytab.
Entry for principal oryx/127.0.98.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/krb5kdc/oryx_127.0.98.66.keytab.
I20260618 22:18:16.332548 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:42267
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/krb5kdc/oryx_127.0.98.66.keytab
--principal=oryx/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:34497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:18:16.438659 6710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:18:16.438894 6710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:18:16.438972 6710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:18:16.442931 6710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:18:16.442978 6710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:18:16.443073 6710 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260618 22:18:16.448354 6710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:34497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/krb5kdc/oryx_127.0.98.66.keytab
--principal=oryx/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:42267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6710
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:18:16.449352 6710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:18:16.450336 6710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:18:16.459643 6718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:16.459638 6715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:16.459707 6710 server_base.cc:1034] running on GCE node
W20260618 22:18:16.459638 6716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:16.460429 6710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:18:16.461023 6710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:18:16.462169 6710 hybrid_clock.cc:648] HybridClock initialized: now 1781821096462158 us; error 24 us; skew 500 ppm
Jun 18 22:18:16 dist-test-slave-g6r4 krb5kdc[6469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821096, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:18:16.464864 6710 init.cc:377] Logged in from keytab as oryx/127.0.98.66@KRBTEST.COM (short username oryx)
I20260618 22:18:16.465643 6710 webserver.cc:460] Webserver started at http://127.0.98.66:46797/ using document root <none> and password file <none>
I20260618 22:18:16.466217 6710 fs_manager.cc:359] Metadata directory not provided
I20260618 22:18:16.466280 6710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:18:16.466436 6710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:18:16.468561 6710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-1/data/instance:
uuid: "d7dc899699d84381a27dbdf8dae05cc2"
format_stamp: "Formatted at 2026-06-18 22:18:16 on dist-test-slave-g6r4"
server_key: "f763b4a3ec406bce7106d4dbc831362c"
server_key_iv: "e1102861e62f775def89b3d48caa7285"
server_key_version: "encryptionkey@0"
I20260618 22:18:16.469103 6710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-1/wal/instance:
uuid: "d7dc899699d84381a27dbdf8dae05cc2"
format_stamp: "Formatted at 2026-06-18 22:18:16 on dist-test-slave-g6r4"
server_key: "f763b4a3ec406bce7106d4dbc831362c"
server_key_iv: "e1102861e62f775def89b3d48caa7285"
server_key_version: "encryptionkey@0"
I20260618 22:18:16.472877 6710 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260618 22:18:16.475128 6725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:16.476297 6710 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260618 22:18:16.476428 6710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-1/wal
uuid: "d7dc899699d84381a27dbdf8dae05cc2"
format_stamp: "Formatted at 2026-06-18 22:18:16 on dist-test-slave-g6r4"
server_key: "f763b4a3ec406bce7106d4dbc831362c"
server_key_iv: "e1102861e62f775def89b3d48caa7285"
server_key_version: "encryptionkey@0"
I20260618 22:18:16.476533 6710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:18:16.498765 6710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:18:16.502065 6710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:18:16.502285 6710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:18:16.502856 6710 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:18:16.503610 6710 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:18:16.503654 6710 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:16.503727 6710 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:18:16.503763 6710 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:16.512406 6710 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:41835
I20260618 22:18:16.513044 6838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:41835 every 8 connection(s)
I20260618 22:18:16.513460 6710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
I20260618 22:18:16.518213 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 6710
I20260618 22:18:16.518309 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-1/wal/instance
I20260618 22:18:16.518579 393 external_mini_cluster.cc:1428] Setting key dd499e89c66a41e45b2cfef1e21b1c06
Jun 18 22:18:16 dist-test-slave-g6r4 krb5kdc[6469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821096, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM
I20260618 22:18:16.527827 6553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:16.515228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:35159 (local address 127.0.98.126:42267)
0618 22:18:16.515361 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:16.515364 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:16.516156 (+ 792us) server_negotiation.cc:407] Connection header received
0618 22:18:16.517563 (+ 1407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:16.517567 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:16.517625 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:16.517701 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:16.519031 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:16.519677 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:16.520864 (+ 1187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:16.521269 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:16.521393 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:16.523968 (+ 2575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:16.523975 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:16.523981 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:16.524012 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:16.525719 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:16.526252 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:16.526258 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:16.526263 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:16.526362 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:16.526702 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:16.526708 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:16.526711 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:16.526952 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:16.527057 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:16.527514 (+ 457us) server_negotiation.cc:299] Negotiation successful
0618 22:18:16.527635 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260618 22:18:16.528491 6841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:16.515584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:42267 (local address 127.0.98.66:35159)
0618 22:18:16.516007 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:16.516046 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:16.517290 (+ 1244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:16.517876 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:16.517883 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:16.518235 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:16.518884 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:16.518899 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:16.519803 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:16.519807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:16.520735 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:16.520744 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:16.521441 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:16.521447 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:16.521667 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:16.522272 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:16.522293 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:16.523796 (+ 1503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:18:16.525839 (+ 2043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:16.525843 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:16.525854 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:16.526151 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:16.526457 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:16.526459 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:16.526461 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:16.526580 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:16.527053 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:18:16.527058 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:18:16.527261 (+ 203us) client_negotiation.cc:769] Sending connection context
0618 22:18:16.527405 (+ 144us) client_negotiation.cc:240] Negotiation successful
0618 22:18:16.527635 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":107,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.67@KRBTEST.COM; defaulting to no policy
I20260618 22:18:16.529605 6839 heartbeater.cc:347] Connected to a master server at 127.0.98.126:42267
I20260618 22:18:16.529847 6839 heartbeater.cc:464] Registering TS with master...
I20260618 22:18:16.530313 6839 heartbeater.cc:511] Master 127.0.98.126:42267 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.67@KRBTEST.COM" created.
I20260618 22:18:16.531340 6514 ts_manager.cc:194] Registered new tserver with Master: d7dc899699d84381a27dbdf8dae05cc2 (127.0.98.66:41835)
I20260618 22:18:16.532146 6514 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.66@KRBTEST.COM'} at 127.0.98.66:35159
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/krb5kdc/oryx_127.0.98.67.keytab.
Entry for principal oryx/127.0.98.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/krb5kdc/oryx_127.0.98.67.keytab.
I20260618 22:18:16.546406 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:42267
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/krb5kdc/oryx_127.0.98.67.keytab
--principal=oryx/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:34497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:18:16.653796 6844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:18:16.654130 6844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:18:16.654222 6844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:18:16.658236 6844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:18:16.658308 6844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:18:16.658391 6844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260618 22:18:16.663511 6844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:34497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/krb5kdc/oryx_127.0.98.67.keytab
--principal=oryx/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:42267
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:18:16.664480 6844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:18:16.665462 6844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:18:16.675282 6850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:16.675341 6849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:16.675302 6852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:16.675565 6844 server_base.cc:1034] running on GCE node
I20260618 22:18:16.675940 6844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:18:16.676528 6844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:18:16.677724 6844 hybrid_clock.cc:648] HybridClock initialized: now 1781821096677692 us; error 45 us; skew 500 ppm
Jun 18 22:18:16 dist-test-slave-g6r4 krb5kdc[6469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821096, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:18:16.680559 6844 init.cc:377] Logged in from keytab as oryx/127.0.98.67@KRBTEST.COM (short username oryx)
I20260618 22:18:16.681383 6844 webserver.cc:460] Webserver started at http://127.0.98.67:40627/ using document root <none> and password file <none>
I20260618 22:18:16.681879 6844 fs_manager.cc:359] Metadata directory not provided
I20260618 22:18:16.681995 6844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:18:16.682157 6844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:18:16.684026 6844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-2/data/instance:
uuid: "5256bda72de544e985aaad69508c7748"
format_stamp: "Formatted at 2026-06-18 22:18:16 on dist-test-slave-g6r4"
server_key: "0e3adbfbc258158b0970603c0bf1fc94"
server_key_iv: "9b5e649e3867a672fadfd64652f7ee60"
server_key_version: "encryptionkey@0"
I20260618 22:18:16.684527 6844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-2/wal/instance:
uuid: "5256bda72de544e985aaad69508c7748"
format_stamp: "Formatted at 2026-06-18 22:18:16 on dist-test-slave-g6r4"
server_key: "0e3adbfbc258158b0970603c0bf1fc94"
server_key_iv: "9b5e649e3867a672fadfd64652f7ee60"
server_key_version: "encryptionkey@0"
I20260618 22:18:16.688162 6844 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260618 22:18:16.690448 6859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:16.691673 6844 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:18:16.691803 6844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-2/wal
uuid: "5256bda72de544e985aaad69508c7748"
format_stamp: "Formatted at 2026-06-18 22:18:16 on dist-test-slave-g6r4"
server_key: "0e3adbfbc258158b0970603c0bf1fc94"
server_key_iv: "9b5e649e3867a672fadfd64652f7ee60"
server_key_version: "encryptionkey@0"
I20260618 22:18:16.691926 6844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:18:16.709096 6844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:18:16.711930 6844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:18:16.712121 6844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:18:16.712663 6844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:18:16.713435 6844 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:18:16.713506 6844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:16.713575 6844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:18:16.713619 6844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:16.722626 6844 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:46203
I20260618 22:18:16.722652 6972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:46203 every 8 connection(s)
I20260618 22:18:16.723495 6844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
I20260618 22:18:16.731796 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 6844
I20260618 22:18:16.731930 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-2/wal/instance
I20260618 22:18:16.732188 393 external_mini_cluster.cc:1428] Setting key 2410f1d1e8723fa1235a4a1621dbd6be
Jun 18 22:18:16 dist-test-slave-g6r4 krb5kdc[6469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821096, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM
I20260618 22:18:16.736836 6553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:16.725172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:45357 (local address 127.0.98.126:42267)
0618 22:18:16.725303 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:16.725307 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:16.726182 (+ 875us) server_negotiation.cc:407] Connection header received
0618 22:18:16.727728 (+ 1546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:16.727732 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:16.727794 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:16.727898 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:16.729033 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:16.729557 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:16.730689 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:16.730917 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:16.731028 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:16.733582 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:16.733585 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:16.733587 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:16.733608 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:16.735128 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:16.735607 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:16.735610 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:16.735611 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:16.735680 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:16.735965 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:16.735967 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:16.735968 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:16.736125 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:16.736199 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:16.736620 (+ 421us) server_negotiation.cc:299] Negotiation successful
0618 22:18:16.736711 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260618 22:18:16.737643 6975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:16.725466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:42267 (local address 127.0.98.67:45357)
0618 22:18:16.726024 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:16.726064 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:16.727359 (+ 1295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:16.728065 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:16.728072 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:16.728446 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:16.728882 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:16.728896 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:16.729682 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:16.729686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:16.730559 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:16.730568 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:16.731060 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:16.731063 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:16.731231 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:16.731783 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:16.731798 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:16.733412 (+ 1614us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:18:16.735251 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:16.735255 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:16.735265 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:16.735518 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:16.735758 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:16.735760 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:16.735762 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:16.735883 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:16.736209 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:18:16.736214 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:18:16.736427 (+ 213us) client_negotiation.cc:769] Sending connection context
0618 22:18:16.736575 (+ 148us) client_negotiation.cc:240] Negotiation successful
0618 22:18:16.736853 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":98,"threads_started":1}
I20260618 22:18:16.738763 6973 heartbeater.cc:347] Connected to a master server at 127.0.98.126:42267
I20260618 22:18:16.738955 6973 heartbeater.cc:464] Registering TS with master...
I20260618 22:18:16.739349 6973 heartbeater.cc:511] Master 127.0.98.126:42267 requested a full tablet report, sending...
I20260618 22:18:16.740242 6514 ts_manager.cc:194] Registered new tserver with Master: 5256bda72de544e985aaad69508c7748 (127.0.98.67:46203)
I20260618 22:18:16.740808 6514 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.67@KRBTEST.COM'} at 127.0.98.67:45357
I20260618 22:18:16.745667 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/krb5kdc/oryx_127.0.98.125.keytab.
Entry for principal oryx/127.0.98.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/krb5kdc/oryx_127.0.98.125.keytab.
I20260618 22:18:16.768674 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.125:34675
--webserver_interface=127.0.98.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/krb5kdc/oryx_127.0.98.125.keytab
--principal=oryx/127.0.98.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:34497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:42267,127.0.98.125:34675
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:18:16.877177 6978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:18:16.877462 6978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:18:16.877575 6978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:18:16.882198 6978 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:18:16.882282 6978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:18:16.882324 6978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:18:16.882392 6978 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:18:16.882437 6978 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:18:16.887931 6978 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:34497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:42267,127.0.98.125:34675
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/krb5kdc/oryx_127.0.98.125.keytab
--principal=oryx/127.0.98.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.125:34675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
W20260618 22:18:16.888152 6978 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.126:42267,127.0.98.125:34675'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260618 22:18:16.888973 6978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:18:16.889993 6978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:18:16.898483 6983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:16.898464 6986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:16.898437 6984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:16.901322 6978 server_base.cc:1034] running on GCE node
I20260618 22:18:16.901978 6978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:18:16.903380 6978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:18:16.904546 6978 hybrid_clock.cc:648] HybridClock initialized: now 1781821096904479 us; error 98 us; skew 500 ppm
Jun 18 22:18:16 dist-test-slave-g6r4 krb5kdc[6469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821096, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:18:16.907407 6978 init.cc:377] Logged in from keytab as oryx/127.0.98.125@KRBTEST.COM (short username oryx)
I20260618 22:18:16.908243 6978 webserver.cc:460] Webserver started at http://127.0.98.125:44771/ using document root <none> and password file <none>
I20260618 22:18:16.908783 6978 fs_manager.cc:359] Metadata directory not provided
I20260618 22:18:16.908876 6978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:18:16.909103 6978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:18:16.911132 6978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/master-1/data/instance:
uuid: "512c63ff46bd41d6a51ca7e026cc8d47"
format_stamp: "Formatted at 2026-06-18 22:18:16 on dist-test-slave-g6r4"
server_key: "061f92662116d1816015d32ff7e0618d"
server_key_iv: "863d541c40e3c19d736086e481ae6fc4"
server_key_version: "encryptionkey@0"
I20260618 22:18:16.911698 6978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/master-1/wal/instance:
uuid: "512c63ff46bd41d6a51ca7e026cc8d47"
format_stamp: "Formatted at 2026-06-18 22:18:16 on dist-test-slave-g6r4"
server_key: "061f92662116d1816015d32ff7e0618d"
server_key_iv: "863d541c40e3c19d736086e481ae6fc4"
server_key_version: "encryptionkey@0"
I20260618 22:18:16.915210 6978 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:18:16.917555 6993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:16.918869 6978 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:18:16.918999 6978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/master-1/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/master-1/wal
uuid: "512c63ff46bd41d6a51ca7e026cc8d47"
format_stamp: "Formatted at 2026-06-18 22:18:16 on dist-test-slave-g6r4"
server_key: "061f92662116d1816015d32ff7e0618d"
server_key_iv: "863d541c40e3c19d736086e481ae6fc4"
server_key_version: "encryptionkey@0"
I20260618 22:18:16.919102 6978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:18:16.950500 6978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:18:16.953701 6978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:18:16.953954 6978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:18:16.960333 6978 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.125:34675
I20260618 22:18:16.960610 7045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.125:34675 every 8 connection(s)
I20260618 22:18:16.961278 6978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/master-1/data/info.pb
I20260618 22:18:16.963855 7046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:18:16.964426 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 6978
I20260618 22:18:16.968529 7046 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42267 } has no permanent_uuid. Determining permanent_uuid...
Jun 18 22:18:16 dist-test-slave-g6r4 krb5kdc[6469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.125@KRBTEST.COM
Jun 18 22:18:16 dist-test-slave-g6r4 krb5kdc[6469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821096, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.125@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM
I20260618 22:18:16.979295 7049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:16.965832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54006 (local address 127.0.98.125:34675)
0618 22:18:16.966261 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:16.966285 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:16.966337 (+ 52us) server_negotiation.cc:407] Connection header received
0618 22:18:16.967011 (+ 674us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:16.967027 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:16.967323 (+ 296us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:16.967577 (+ 254us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:16.968427 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:16.969453 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:16.970599 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:16.971231 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:16.971433 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:16.974067 (+ 2634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:16.974072 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:16.974081 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:16.974105 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:16.975947 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:16.976443 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:16.976451 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:16.976458 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:16.976593 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:16.976969 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:16.976971 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:16.976972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:16.977463 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:16.977599 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:16.978031 (+ 432us) server_negotiation.cc:299] Negotiation successful
0618 22:18:16.978290 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"spinlock_wait_cycles":5504,"thread_start_us":127,"threads_started":1}
I20260618 22:18:16.979528 7050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:16.970172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:42267 (local address 127.0.0.1:59694)
0618 22:18:16.970532 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:16.970577 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:16.970691 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:16.971138 (+ 447us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:16.971142 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:16.971510 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:16.971626 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:16.971632 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:16.972775 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:16.972778 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:16.973413 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:16.973421 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:16.973957 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:16.973961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:16.974078 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:16.974590 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:16.974635 (+ 45us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:16.976040 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:18:16.977934 (+ 1894us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:16.977948 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:16.977953 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:16.978207 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:16.978467 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:16.978470 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:16.978472 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:16.978559 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:16.978944 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:18:16.978951 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:18:16.979055 (+ 104us) client_negotiation.cc:769] Sending connection context
0618 22:18:16.979195 (+ 140us) client_negotiation.cc:240] Negotiation successful
0618 22:18:16.979355 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":140,"threads_started":1}
I20260618 22:18:16.979542 6553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:16.970233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59694 (local address 127.0.98.126:42267)
0618 22:18:16.970366 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:16.970371 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:16.970661 (+ 290us) server_negotiation.cc:407] Connection header received
0618 22:18:16.970841 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:16.970848 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:16.970906 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:16.970952 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:16.971765 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:16.972597 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:16.973534 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:16.973793 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:16.973907 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:16.976154 (+ 2247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:16.976157 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:16.976160 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:16.976196 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:16.977807 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:16.978305 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:16.978307 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:16.978308 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:16.978380 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:16.978639 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:16.978641 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:16.978642 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:16.978846 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:16.978921 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:16.979252 (+ 331us) server_negotiation.cc:299] Negotiation successful
0618 22:18:16.979340 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260618 22:18:16.980654 7046 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.125" port: 34675 } has no permanent_uuid. Determining permanent_uuid...
Jun 18 22:18:16 dist-test-slave-g6r4 krb5kdc[6469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821096, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.125@KRBTEST.COM for oryx/127.0.98.125@KRBTEST.COM
I20260618 22:18:16.988159 7050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:16.981082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:34675 (local address 127.0.0.1:54022)
0618 22:18:16.981213 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:16.981227 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:16.981300 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:16.981500 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:16.981502 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:16.981652 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:16.981730 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:16.981734 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:16.982822 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:16.982825 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:16.983316 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:16.983322 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:16.983752 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:16.983755 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:16.983835 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:16.984190 (+ 355us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:16.984200 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:16.985375 (+ 1175us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:18:16.986958 (+ 1583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:16.986961 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:16.986962 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:16.987133 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:16.987352 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:16.987354 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:16.987355 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:16.987420 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:16.987744 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:18:16.987746 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:18:16.987818 (+ 72us) client_negotiation.cc:769] Sending connection context
0618 22:18:16.987912 (+ 94us) client_negotiation.cc:240] Negotiation successful
0618 22:18:16.988003 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260618 22:18:16.988169 7049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:16.981181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54022 (local address 127.0.98.125:34675)
0618 22:18:16.981269 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:16.981273 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:16.981286 (+ 13us) server_negotiation.cc:407] Connection header received
0618 22:18:16.981368 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:16.981373 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:16.981406 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:16.981492 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:16.981836 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:16.982703 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:16.983423 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:16.983649 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:16.983755 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:16.985494 (+ 1739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:16.985496 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:16.985498 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:16.985518 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:16.986867 (+ 1349us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:16.987213 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:16.987215 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:16.987216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:16.987279 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:16.987493 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:16.987495 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:16.987496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:16.987630 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:16.987699 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:16.987960 (+ 261us) server_negotiation.cc:299] Negotiation successful
0618 22:18:16.988047 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":22}
I20260618 22:18:16.991031 7046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 512c63ff46bd41d6a51ca7e026cc8d47: Bootstrap starting.
I20260618 22:18:16.993084 7046 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 512c63ff46bd41d6a51ca7e026cc8d47: Neither blocks nor log segments found. Creating new log.
I20260618 22:18:16.993819 7046 log.cc:824] T 00000000000000000000000000000000 P 512c63ff46bd41d6a51ca7e026cc8d47: Log is configured to *not* fsync() on all Append() calls
I20260618 22:18:16.995622 7046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 512c63ff46bd41d6a51ca7e026cc8d47: No bootstrap required, opened a new log
I20260618 22:18:16.998531 7046 raft_consensus.cc:348] T 00000000000000000000000000000000 P 512c63ff46bd41d6a51ca7e026cc8d47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cafe178b94854b17acaae5da06c1920d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42267 } } peers { permanent_uuid: "512c63ff46bd41d6a51ca7e026cc8d47" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 34675 } }
I20260618 22:18:16.998705 7046 raft_consensus.cc:374] T 00000000000000000000000000000000 P 512c63ff46bd41d6a51ca7e026cc8d47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:18:16.998785 7046 raft_consensus.cc:729] T 00000000000000000000000000000000 P 512c63ff46bd41d6a51ca7e026cc8d47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 512c63ff46bd41d6a51ca7e026cc8d47, State: Initialized, Role: FOLLOWER
I20260618 22:18:16.999179 7046 consensus_queue.cc:260] T 00000000000000000000000000000000 P 512c63ff46bd41d6a51ca7e026cc8d47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cafe178b94854b17acaae5da06c1920d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42267 } } peers { permanent_uuid: "512c63ff46bd41d6a51ca7e026cc8d47" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 34675 } }
I20260618 22:18:16.999598 7052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 512c63ff46bd41d6a51ca7e026cc8d47 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cafe178b94854b17acaae5da06c1920d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42267 } } peers { permanent_uuid: "512c63ff46bd41d6a51ca7e026cc8d47" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 34675 } } }
I20260618 22:18:16.999723 7052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 512c63ff46bd41d6a51ca7e026cc8d47 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:18:17.000279 7046 sys_catalog.cc:564] T 00000000000000000000000000000000 P 512c63ff46bd41d6a51ca7e026cc8d47 [sys.catalog]: configured and running, proceeding with master startup.
W20260618 22:18:17.005541 7063 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 512c63ff46bd41d6a51ca7e026cc8d47: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 22:18:17.005623 7063 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 22:18:17.006412 6978 master_runner.cc:428] Detected that this master 512c63ff46bd41d6a51ca7e026cc8d47 is joining an existing cluster
I20260618 22:18:17.006474 6978 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.125:34675
I20260618 22:18:17.016882 7049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:17.008752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54038 (local address 127.0.98.125:34675)
0618 22:18:17.008861 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:17.008865 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:17.009060 (+ 195us) server_negotiation.cc:407] Connection header received
0618 22:18:17.009136 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:17.009140 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:17.009176 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:17.009254 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:17.009806 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.010528 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.011669 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.011877 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.011986 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:17.013239 (+ 1253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:17.013242 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:17.013245 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:17.013268 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:17.015129 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:17.015642 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:17.015645 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:17.015647 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:17.015739 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:17.016045 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:17.016049 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:17.016050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:17.016243 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:17.016335 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:17.016590 (+ 255us) server_negotiation.cc:299] Negotiation successful
0618 22:18:17.016728 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":41}
I20260618 22:18:17.017261 7070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:17.008650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:34675 (local address 127.0.0.1:54038)
0618 22:18:17.008956 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:17.008969 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:17.009062 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:17.009301 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:17.009304 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:17.009562 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:17.009686 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:17.009693 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.010641 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.010645 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:17.011551 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:17.011561 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.011994 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.011997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:17.012167 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:17.012654 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:17.012667 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:17.013107 (+ 440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:18:17.015241 (+ 2134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:17.015244 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:17.015247 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:17.015518 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:17.015846 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:17.015851 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:17.015853 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:17.015946 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:17.016367 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:18:17.016369 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:18:17.016444 (+ 75us) client_negotiation.cc:769] Sending connection context
0618 22:18:17.016997 (+ 553us) client_negotiation.cc:240] Negotiation successful
0618 22:18:17.017109 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":40,"thread_start_us":98,"threads_started":1}
I20260618 22:18:17.019145 7069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:17.008252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:42267 (local address 127.0.0.1:59700)
0618 22:18:17.008579 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:17.008612 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:17.008746 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:17.009150 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:17.009153 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:17.009431 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:17.009517 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:17.009523 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.011685 (+ 2162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.011688 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:17.012693 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:17.012714 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.013278 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.013281 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:17.013402 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:17.014515 (+ 1113us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:17.014534 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:17.014964 (+ 430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:18:17.017230 (+ 2266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:17.017232 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:17.017234 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:17.017777 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:17.018111 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:17.018116 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:17.018118 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:17.018203 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:17.018597 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:18:17.018601 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:18:17.018712 (+ 111us) client_negotiation.cc:769] Sending connection context
0618 22:18:17.018847 (+ 135us) client_negotiation.cc:240] Negotiation successful
0618 22:18:17.018975 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":31,"thread_start_us":92,"threads_started":1}
I20260618 22:18:17.019171 6553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:17.008334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59700 (local address 127.0.98.126:42267)
0618 22:18:17.008460 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:17.008463 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:17.008678 (+ 215us) server_negotiation.cc:407] Connection header received
0618 22:18:17.008880 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:17.008884 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:17.008935 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:17.008975 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:17.009648 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.010517 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.012836 (+ 2319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.013187 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.013281 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:17.015096 (+ 1815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:17.015100 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:17.015103 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:17.015129 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:17.016859 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:17.017879 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:17.017883 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:17.017885 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:17.018001 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:17.018318 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:17.018322 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:17.018323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:17.018492 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:17.018596 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:17.018901 (+ 305us) server_negotiation.cc:299] Negotiation successful
0618 22:18:17.019032 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
Jun 18 22:18:17 dist-test-slave-g6r4 krb5kdc[6469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821095, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.126@KRBTEST.COM for oryx/127.0.98.125@KRBTEST.COM
I20260618 22:18:17.031960 7049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:17.023599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54042 (local address 127.0.98.125:34675)
0618 22:18:17.023696 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:17.023700 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:17.024003 (+ 303us) server_negotiation.cc:407] Connection header received
0618 22:18:17.024171 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:17.024174 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:17.024220 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:17.024303 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:17.024816 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.025549 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.026416 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.026745 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.026843 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:17.028914 (+ 2071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:17.028917 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:17.028920 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:17.028942 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:17.030406 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:17.030812 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:17.030816 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:17.030817 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:17.030897 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:17.031133 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:17.031136 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:17.031138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:17.031322 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:17.031421 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:17.031721 (+ 300us) server_negotiation.cc:299] Negotiation successful
0618 22:18:17.031820 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260618 22:18:17.032020 7071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:17.023599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:34675 (local address 127.0.0.1:54042)
0618 22:18:17.023885 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:17.023931 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:17.024051 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:17.024323 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:17.024327 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:17.024625 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:17.024715 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:17.024720 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.025648 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.025651 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:17.026314 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:17.026323 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.026855 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.026858 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:17.026929 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:17.027432 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:17.027444 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:17.028771 (+ 1327us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:18:17.030504 (+ 1733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:17.030511 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:17.030515 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:17.030724 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:17.030982 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:17.030984 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:17.030985 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:17.031055 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:17.031416 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:18:17.031421 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:18:17.031516 (+ 95us) client_negotiation.cc:769] Sending connection context
0618 22:18:17.031653 (+ 137us) client_negotiation.cc:240] Negotiation successful
0618 22:18:17.031840 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":67,"threads_started":1}
I20260618 22:18:17.033334 6514 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.125:34675: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "512c63ff46bd41d6a51ca7e026cc8d47"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.125"
port: 34675
}
attrs {
promote: true
}
}
dest_uuid: "cafe178b94854b17acaae5da06c1920d"
cas_config_opid_index: -1
I20260618 22:18:17.033982 6514 consensus_queue.cc:237] T 00000000000000000000000000000000 P cafe178b94854b17acaae5da06c1920d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "cafe178b94854b17acaae5da06c1920d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42267 } } peers { permanent_uuid: "512c63ff46bd41d6a51ca7e026cc8d47" member_type: NON_VOTER last_known_addr { host: "127.0.98.125" port: 34675 } attrs { promote: true } }
I20260618 22:18:17.035720 7073 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cafe178b94854b17acaae5da06c1920d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 512c63ff46bd41d6a51ca7e026cc8d47 (127.0.98.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "cafe178b94854b17acaae5da06c1920d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42267 } } peers { permanent_uuid: "512c63ff46bd41d6a51ca7e026cc8d47" member_type: NON_VOTER last_known_addr { host: "127.0.98.125" port: 34675 } attrs { promote: true } } }
I20260618 22:18:17.036504 7020 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 512c63ff46bd41d6a51ca7e026cc8d47 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:18:17.037388 7073 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.125:34675
I20260618 22:18:17.037814 6978 master.cc:537] Master@127.0.98.125:34675 shutting down...
I20260618 22:18:17.037953 7020 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 512c63ff46bd41d6a51ca7e026cc8d47 [term 1 FOLLOWER]: Refusing update from remote peer cafe178b94854b17acaae5da06c1920d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260618 22:18:17.038761 7075 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cafe178b94854b17acaae5da06c1920d [LEADER]: Connected to new peer: Peer: permanent_uuid: "512c63ff46bd41d6a51ca7e026cc8d47" member_type: NON_VOTER last_known_addr { host: "127.0.98.125" port: 34675 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260618 22:18:17.040846 6498 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260618 22:18:17.042521 6498 consensus_peers.cc:487] T 00000000000000000000000000000000 P cafe178b94854b17acaae5da06c1920d -> Peer 512c63ff46bd41d6a51ca7e026cc8d47 (127.0.98.125:34675): Couldn't send request to peer 512c63ff46bd41d6a51ca7e026cc8d47. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260618 22:18:17.043208 6978 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 512c63ff46bd41d6a51ca7e026cc8d47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:18:17.043465 6978 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 512c63ff46bd41d6a51ca7e026cc8d47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:18:17.043561 6978 tablet_replica.cc:330] T 00000000000000000000000000000000 P 512c63ff46bd41d6a51ca7e026cc8d47: stopping tablet replica
WARNING: no policy specified for oryx/127.0.98.124@KRBTEST.COM; defaulting to no policy
I20260618 22:18:17.044642 7074 sys_catalog.cc:455] T 00000000000000000000000000000000 P cafe178b94854b17acaae5da06c1920d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "cafe178b94854b17acaae5da06c1920d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "cafe178b94854b17acaae5da06c1920d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42267 } } peers { permanent_uuid: "512c63ff46bd41d6a51ca7e026cc8d47" member_type: NON_VOTER last_known_addr { host: "127.0.98.125" port: 34675 } attrs { promote: true } } }
I20260618 22:18:17.044754 7074 sys_catalog.cc:458] T 00000000000000000000000000000000 P cafe178b94854b17acaae5da06c1920d [sys.catalog]: This master's current role is: LEADER
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.124@KRBTEST.COM" created.
I20260618 22:18:17.048496 6978 master.cc:552] Master@127.0.98.125:34675 shutdown complete.
I20260618 22:18:17.048588 6978 master_runner.cc:305] Clearing existing system tablet
I20260618 22:18:17.048919 6978 fs_manager.cc:359] Metadata directory not provided
I20260618 22:18:17.049017 6978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:18:17.050228 6978 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260618 22:18:17.051025 7080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:17.051220 6978 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:18:17.051371 6978 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260618 22:18:17.051409 6978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/master-1/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/master-1/wal
uuid: "512c63ff46bd41d6a51ca7e026cc8d47"
format_stamp: "Formatted at 2026-06-18 22:18:16 on dist-test-slave-g6r4"
server_key: "061f92662116d1816015d32ff7e0618d"
server_key_iv: "863d541c40e3c19d736086e481ae6fc4"
server_key_version: "encryptionkey@0"
I20260618 22:18:17.053342 6978 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 512c63ff46bd41d6a51ca7e026cc8d47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:18:17.056053 6978 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 512c63ff46bd41d6a51ca7e026cc8d47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260618 22:18:17.056145 6978 log.cc:1196] T 00000000000000000000000000000000 P 512c63ff46bd41d6a51ca7e026cc8d47: Deleting WAL directory at /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260618 22:18:17.056417 6978 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 512c63ff46bd41d6a51ca7e026cc8d47: Deleting consensus metadata
I20260618 22:18:17.056589 6978 master_runner.cc:315] Copying system tablet from 127.0.98.126:42267
I20260618 22:18:17.057664 6978 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 512c63ff46bd41d6a51ca7e026cc8d47: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.126:42267
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/krb5kdc/oryx_127.0.98.124.keytab.
Entry for principal oryx/127.0.98.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/krb5kdc/oryx_127.0.98.124.keytab.
I20260618 22:18:17.070627 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.124:36495
--webserver_interface=127.0.98.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/krb5kdc/oryx_127.0.98.124.keytab
--principal=oryx/127.0.98.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:34497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:42267,127.0.98.125:34675,127.0.98.124:36495
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 22:18:17.072331 6553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:17.058247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59712 (local address 127.0.98.126:42267)
0618 22:18:17.058379 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:17.058384 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:17.063537 (+ 5153us) server_negotiation.cc:407] Connection header received
0618 22:18:17.063666 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:17.063670 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:17.063733 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:17.063823 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:17.064420 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.065252 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.066529 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.066976 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.067137 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:17.068600 (+ 1463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:17.068605 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:17.068608 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:17.068636 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:17.070427 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:17.070955 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:17.070959 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:17.070961 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:17.071052 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:17.071426 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:17.071430 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:17.071432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:17.071653 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:17.071741 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:17.072129 (+ 388us) server_negotiation.cc:299] Negotiation successful
0618 22:18:17.072222 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260618 22:18:17.072429 7086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:17.058194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:42267 (local address 127.0.0.1:59712)
0618 22:18:17.063415 (+ 5221us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:17.063431 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:17.063555 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:17.063933 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:17.063936 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:17.064133 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:17.064289 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:17.064296 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.065385 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.065389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:17.066406 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:17.066415 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.067112 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.067115 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:17.067225 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:17.067963 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:17.067977 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:17.068428 (+ 451us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:18:17.070558 (+ 2130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:17.070561 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:17.070564 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:17.070838 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:17.071178 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:17.071182 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:17.071184 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:17.071305 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:17.071817 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:18:17.071820 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:18:17.071916 (+ 96us) client_negotiation.cc:769] Sending connection context
0618 22:18:17.072068 (+ 152us) client_negotiation.cc:240] Negotiation successful
0618 22:18:17.072216 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5148,"thread_start_us":87,"threads_started":1}
I20260618 22:18:17.072924 6534 tablet_copy_service.cc:140] P cafe178b94854b17acaae5da06c1920d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 512c63ff46bd41d6a51ca7e026cc8d47 ({username='oryx', principal='oryx/127.0.98.125@KRBTEST.COM'} at 127.0.0.1:59712)
I20260618 22:18:17.073092 6534 tablet_copy_service.cc:161] P cafe178b94854b17acaae5da06c1920d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 512c63ff46bd41d6a51ca7e026cc8d47 at {username='oryx', principal='oryx/127.0.98.125@KRBTEST.COM'} at 127.0.0.1:59712: session id = 512c63ff46bd41d6a51ca7e026cc8d47-00000000000000000000000000000000
I20260618 22:18:17.074661 6534 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P cafe178b94854b17acaae5da06c1920d: Tablet Copy: opened 0 blocks and 1 log segments
I20260618 22:18:17.076179 6978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260618 22:18:17.078317 6978 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 512c63ff46bd41d6a51ca7e026cc8d47: tablet copy: Starting download of 0 data blocks...
I20260618 22:18:17.078495 6978 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 512c63ff46bd41d6a51ca7e026cc8d47: tablet copy: Starting download of 1 WAL segments...
I20260618 22:18:17.079984 6978 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 512c63ff46bd41d6a51ca7e026cc8d47: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260618 22:18:17.081532 6534 tablet_copy_service.cc:342] P cafe178b94854b17acaae5da06c1920d: Request end of tablet copy session 512c63ff46bd41d6a51ca7e026cc8d47-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.125@KRBTEST.COM'} at 127.0.0.1:59712
I20260618 22:18:17.081645 6534 tablet_copy_service.cc:433] P cafe178b94854b17acaae5da06c1920d: ending tablet copy session 512c63ff46bd41d6a51ca7e026cc8d47-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 512c63ff46bd41d6a51ca7e026cc8d47
I20260618 22:18:17.088017 6978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:18:17.089460 7088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:17.094871 7089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:17.094888 7091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:17.095180 6978 server_base.cc:1034] running on GCE node
I20260618 22:18:17.095341 6978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:18:17.095585 6978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:18:17.096709 6978 hybrid_clock.cc:648] HybridClock initialized: now 1781821097096696 us; error 29 us; skew 500 ppm
Jun 18 22:18:17 dist-test-slave-g6r4 krb5kdc[6469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821097, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:18:17.098165 6978 init.cc:377] Logged in from keytab as oryx/127.0.98.125@KRBTEST.COM (short username oryx)
I20260618 22:18:17.098623 6978 webserver.cc:460] Webserver started at http://127.0.98.125:45923/ using document root <none> and password file <none>
I20260618 22:18:17.098804 6978 fs_manager.cc:359] Metadata directory not provided
I20260618 22:18:17.098862 6978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:18:17.099903 6978 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260618 22:18:17.100816 7098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:17.101038 6978 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 22:18:17.101131 6978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/master-1/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/master-1/wal
uuid: "512c63ff46bd41d6a51ca7e026cc8d47"
format_stamp: "Formatted at 2026-06-18 22:18:16 on dist-test-slave-g6r4"
server_key: "061f92662116d1816015d32ff7e0618d"
server_key_iv: "863d541c40e3c19d736086e481ae6fc4"
server_key_version: "encryptionkey@0"
I20260618 22:18:17.101212 6978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:18:17.108381 6978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:18:17.110355 6978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:18:17.115054 6978 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.125:34675
I20260618 22:18:17.115566 6978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/master-1/data/info.pb
I20260618 22:18:17.117163 7151 sys_catalog.cc:263] Verifying existing consensus state
I20260618 22:18:17.118062 7150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.125:34675 every 8 connection(s)
I20260618 22:18:17.118245 7151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 512c63ff46bd41d6a51ca7e026cc8d47: Bootstrap starting.
I20260618 22:18:17.126013 7151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 512c63ff46bd41d6a51ca7e026cc8d47: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260618 22:18:17.126276 7151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 512c63ff46bd41d6a51ca7e026cc8d47: Bootstrap complete.
I20260618 22:18:17.126724 7151 raft_consensus.cc:348] T 00000000000000000000000000000000 P 512c63ff46bd41d6a51ca7e026cc8d47 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "cafe178b94854b17acaae5da06c1920d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42267 } } peers { permanent_uuid: "512c63ff46bd41d6a51ca7e026cc8d47" member_type: NON_VOTER last_known_addr { host: "127.0.98.125" port: 34675 } attrs { promote: true } }
I20260618 22:18:17.126854 7151 raft_consensus.cc:729] T 00000000000000000000000000000000 P 512c63ff46bd41d6a51ca7e026cc8d47 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 512c63ff46bd41d6a51ca7e026cc8d47, State: Initialized, Role: LEARNER
I20260618 22:18:17.126966 7151 consensus_queue.cc:260] T 00000000000000000000000000000000 P 512c63ff46bd41d6a51ca7e026cc8d47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "cafe178b94854b17acaae5da06c1920d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42267 } } peers { permanent_uuid: "512c63ff46bd41d6a51ca7e026cc8d47" member_type: NON_VOTER last_known_addr { host: "127.0.98.125" port: 34675 } attrs { promote: true } }
I20260618 22:18:17.127282 7154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 512c63ff46bd41d6a51ca7e026cc8d47 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "cafe178b94854b17acaae5da06c1920d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42267 } } peers { permanent_uuid: "512c63ff46bd41d6a51ca7e026cc8d47" member_type: NON_VOTER last_known_addr { host: "127.0.98.125" port: 34675 } attrs { promote: true } } }
I20260618 22:18:17.127440 7154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 512c63ff46bd41d6a51ca7e026cc8d47 [sys.catalog]: This master's current role is: LEARNER
I20260618 22:18:17.127457 7151 sys_catalog.cc:564] T 00000000000000000000000000000000 P 512c63ff46bd41d6a51ca7e026cc8d47 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:18:17.130205 7165 catalog_manager.cc:1259] Loaded cluster ID: fc2c6ba654e446178226c06313f3379d
I20260618 22:18:17.130471 7165 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 512c63ff46bd41d6a51ca7e026cc8d47: loading cluster ID for follower catalog manager: success
I20260618 22:18:17.131971 7165 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 512c63ff46bd41d6a51ca7e026cc8d47: acquiring CA information for follower catalog manager: success
I20260618 22:18:17.133014 7165 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 512c63ff46bd41d6a51ca7e026cc8d47: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260618 22:18:17.220078 7087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:18:17.220345 7087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:18:17.220386 7087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:18:17.224380 7087 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:18:17.224480 7087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:18:17.224510 7087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:18:17.224534 7087 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:18:17.224558 7087 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:18:17.230253 7087 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:34497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:42267,127.0.98.125:34675,127.0.98.124:36495
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/krb5kdc/oryx_127.0.98.124.keytab
--principal=oryx/127.0.98.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.124:36495
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:18:17.231194 7087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:18:17.232131 7087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:18:17.240586 7171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:17.240609 7170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:17.240597 7173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:17.240881 7087 server_base.cc:1034] running on GCE node
I20260618 22:18:17.241545 7087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:18:17.242630 7087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:18:17.243800 7087 hybrid_clock.cc:648] HybridClock initialized: now 1781821097243770 us; error 43 us; skew 500 ppm
Jun 18 22:18:17 dist-test-slave-g6r4 krb5kdc[6469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821097, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:18:17.246553 7087 init.cc:377] Logged in from keytab as oryx/127.0.98.124@KRBTEST.COM (short username oryx)
I20260618 22:18:17.247398 7087 webserver.cc:460] Webserver started at http://127.0.98.124:40083/ using document root <none> and password file <none>
I20260618 22:18:17.247926 7087 fs_manager.cc:359] Metadata directory not provided
I20260618 22:18:17.247985 7087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:18:17.248224 7087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:18:17.250250 7087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/master-2/data/instance:
uuid: "9d8bc0e14290456db37208ee7d35b1cc"
format_stamp: "Formatted at 2026-06-18 22:18:17 on dist-test-slave-g6r4"
server_key: "e2b6e620a7cd5c75a5639ab2b59b4df7"
server_key_iv: "009abac25c6c03286a2d189fa1c8c7af"
server_key_version: "encryptionkey@0"
I20260618 22:18:17.250794 7087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/master-2/wal/instance:
uuid: "9d8bc0e14290456db37208ee7d35b1cc"
format_stamp: "Formatted at 2026-06-18 22:18:17 on dist-test-slave-g6r4"
server_key: "e2b6e620a7cd5c75a5639ab2b59b4df7"
server_key_iv: "009abac25c6c03286a2d189fa1c8c7af"
server_key_version: "encryptionkey@0"
I20260618 22:18:17.254276 7087 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:18:17.256631 7180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:17.257859 7087 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:18:17.258040 7087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/master-2/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/master-2/wal
uuid: "9d8bc0e14290456db37208ee7d35b1cc"
format_stamp: "Formatted at 2026-06-18 22:18:17 on dist-test-slave-g6r4"
server_key: "e2b6e620a7cd5c75a5639ab2b59b4df7"
server_key_iv: "009abac25c6c03286a2d189fa1c8c7af"
server_key_version: "encryptionkey@0"
I20260618 22:18:17.258145 7087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:18:17.288234 7087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:18:17.291152 7087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:18:17.291356 7087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:18:17.298401 7087 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.124:36495
I20260618 22:18:17.298411 7232 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.124:36495 every 8 connection(s)
I20260618 22:18:17.299340 7087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/master-2/data/info.pb
I20260618 22:18:17.300400 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 7087
I20260618 22:18:17.302582 7233 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:18:17.306703 7233 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42267 } has no permanent_uuid. Determining permanent_uuid...
Jun 18 22:18:17 dist-test-slave-g6r4 krb5kdc[6469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821095, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.124@KRBTEST.COM
Jun 18 22:18:17 dist-test-slave-g6r4 krb5kdc[6469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821097, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.124@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM
I20260618 22:18:17.315503 7235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:17.302136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36402 (local address 127.0.98.124:36495)
0618 22:18:17.302574 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:17.302591 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:17.302686 (+ 95us) server_negotiation.cc:407] Connection header received
0618 22:18:17.303314 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:17.303328 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:17.303637 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:17.303970 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:17.304729 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.305864 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.306939 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.307482 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.307662 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:17.310629 (+ 2967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:17.310634 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:17.310642 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:17.310667 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:17.312355 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:17.312879 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:17.312883 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:17.312887 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:17.312992 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:17.313325 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:17.313328 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:17.313330 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:17.313807 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:17.314075 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:17.314345 (+ 270us) server_negotiation.cc:299] Negotiation successful
0618 22:18:17.314592 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"spinlock_wait_cycles":1408,"thread_start_us":138,"threads_started":1}
I20260618 22:18:17.318084 7236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:17.308255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:42267 (local address 127.0.0.1:59728)
0618 22:18:17.309072 (+ 817us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:17.309107 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:17.309202 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:17.309498 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:17.309504 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:17.309969 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:17.310151 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:17.310159 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.310901 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.310905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:17.311552 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:17.311561 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.312048 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.312052 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:17.312173 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:17.312663 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:17.312678 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:17.314345 (+ 1667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:18:17.316421 (+ 2076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:17.316426 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:17.316430 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:17.316687 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:17.316996 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:17.316999 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:17.317000 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:17.317082 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:17.317471 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:18:17.317476 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:18:17.317586 (+ 110us) client_negotiation.cc:769] Sending connection context
0618 22:18:17.317742 (+ 156us) client_negotiation.cc:240] Negotiation successful
0618 22:18:17.317943 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":699,"spinlock_wait_cycles":1408,"thread_start_us":126,"threads_started":1}
I20260618 22:18:17.318081 6553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:17.308395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59728 (local address 127.0.98.126:42267)
0618 22:18:17.308533 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:17.308538 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:17.309189 (+ 651us) server_negotiation.cc:407] Connection header received
0618 22:18:17.309276 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:17.309282 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:17.309346 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:17.309436 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:17.310266 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.310762 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.311717 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.311941 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.312050 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:17.314537 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:17.314540 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:17.314542 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:17.314565 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:17.316315 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:17.316800 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:17.316808 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:17.316821 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:17.316894 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:17.317193 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:17.317197 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:17.317198 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:17.317368 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:17.317475 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:17.317806 (+ 331us) server_negotiation.cc:299] Negotiation successful
0618 22:18:17.317971 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260618 22:18:17.319090 7233 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.125" port: 34675 } has no permanent_uuid. Determining permanent_uuid...
I20260618 22:18:17.322546 6705 heartbeater.cc:503] Master 127.0.98.126:42267 was elected leader, sending a full tablet report...
Jun 18 22:18:17 dist-test-slave-g6r4 krb5kdc[6469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821097, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.124@KRBTEST.COM for oryx/127.0.98.125@KRBTEST.COM
I20260618 22:18:17.326912 7236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:17.319508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:34675 (local address 127.0.0.1:54048)
0618 22:18:17.319608 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:17.319624 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:17.319693 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:17.320072 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:17.320075 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:17.320189 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:17.320266 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:17.320270 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.321177 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.321179 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:17.321686 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:17.321692 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.322148 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.322150 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:17.322214 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:17.322587 (+ 373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:17.322600 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:17.324004 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:18:17.325783 (+ 1779us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:17.325785 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:17.325787 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:17.326023 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:17.326243 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:17.326245 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:17.326246 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:17.326323 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:17.326584 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:18:17.326586 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:18:17.326651 (+ 65us) client_negotiation.cc:769] Sending connection context
0618 22:18:17.326724 (+ 73us) client_negotiation.cc:240] Negotiation successful
0618 22:18:17.326810 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260618 22:18:17.326989 7237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:17.319633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54048 (local address 127.0.98.125:34675)
0618 22:18:17.319885 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:17.319888 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:17.319900 (+ 12us) server_negotiation.cc:407] Connection header received
0618 22:18:17.319935 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:17.319936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:17.319972 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:17.320058 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:17.320395 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.321078 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.321778 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.322056 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.322141 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:17.324145 (+ 2004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:17.324148 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:17.324150 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:17.324171 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:17.325691 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:17.326097 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:17.326099 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:17.326100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:17.326170 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:17.326378 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:17.326379 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:17.326380 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:17.326516 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:17.326580 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:17.326753 (+ 173us) server_negotiation.cc:299] Negotiation successful
0618 22:18:17.326842 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":102,"threads_started":1}
I20260618 22:18:17.327616 7233 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.124" port: 36495 } has no permanent_uuid. Determining permanent_uuid...
Jun 18 22:18:17 dist-test-slave-g6r4 krb5kdc[6469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821097, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.124@KRBTEST.COM for oryx/127.0.98.124@KRBTEST.COM
I20260618 22:18:17.335106 7236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:17.328041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:36495 (local address 127.0.0.1:36412)
0618 22:18:17.328539 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:17.328555 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:17.328635 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:17.328852 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:17.328854 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:17.328967 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:17.329027 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:17.329031 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.329799 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.329801 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:17.330338 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:17.330346 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.330770 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.330772 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:17.330827 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:17.331169 (+ 342us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:17.331179 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:17.332267 (+ 1088us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:18:17.333873 (+ 1606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:17.333875 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:17.333876 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:17.334110 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:17.334365 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:17.334367 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:17.334369 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:17.334434 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:17.334766 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:18:17.334768 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:18:17.334832 (+ 64us) client_negotiation.cc:769] Sending connection context
0618 22:18:17.334905 (+ 73us) client_negotiation.cc:240] Negotiation successful
0618 22:18:17.334985 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260618 22:18:17.335172 7235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:17.328208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36412 (local address 127.0.98.124:36495)
0618 22:18:17.328325 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:17.328328 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:17.328603 (+ 275us) server_negotiation.cc:407] Connection header received
0618 22:18:17.328726 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:17.328728 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:17.328763 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:17.328828 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:17.329126 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.329702 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.330433 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.330681 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.330758 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:17.332378 (+ 1620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:17.332380 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:17.332382 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:17.332401 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:17.333788 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:17.334211 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:17.334214 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:17.334215 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:17.334284 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:17.334508 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:17.334510 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:17.334511 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:17.334690 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:17.334765 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:17.334935 (+ 170us) server_negotiation.cc:299] Negotiation successful
0618 22:18:17.335038 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260618 22:18:17.337795 7233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc: Bootstrap starting.
I20260618 22:18:17.339947 7233 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc: Neither blocks nor log segments found. Creating new log.
I20260618 22:18:17.340654 7233 log.cc:824] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc: Log is configured to *not* fsync() on all Append() calls
I20260618 22:18:17.342428 7233 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc: No bootstrap required, opened a new log
I20260618 22:18:17.345180 7233 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cafe178b94854b17acaae5da06c1920d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42267 } } peers { permanent_uuid: "512c63ff46bd41d6a51ca7e026cc8d47" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 34675 } } peers { permanent_uuid: "9d8bc0e14290456db37208ee7d35b1cc" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 36495 } }
I20260618 22:18:17.345402 7233 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:18:17.345487 7233 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d8bc0e14290456db37208ee7d35b1cc, State: Initialized, Role: FOLLOWER
I20260618 22:18:17.345949 7233 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cafe178b94854b17acaae5da06c1920d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42267 } } peers { permanent_uuid: "512c63ff46bd41d6a51ca7e026cc8d47" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 34675 } } peers { permanent_uuid: "9d8bc0e14290456db37208ee7d35b1cc" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 36495 } }
I20260618 22:18:17.346398 7239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cafe178b94854b17acaae5da06c1920d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42267 } } peers { permanent_uuid: "512c63ff46bd41d6a51ca7e026cc8d47" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 34675 } } peers { permanent_uuid: "9d8bc0e14290456db37208ee7d35b1cc" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 36495 } } }
I20260618 22:18:17.346503 7239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:18:17.347059 7233 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc [sys.catalog]: configured and running, proceeding with master startup.
W20260618 22:18:17.351290 7250 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 22:18:17.351375 7250 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 22:18:17.355064 7087 master_runner.cc:428] Detected that this master 9d8bc0e14290456db37208ee7d35b1cc is joining an existing cluster
I20260618 22:18:17.355123 7087 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.124:36495
I20260618 22:18:17.365872 7237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:17.357370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54054 (local address 127.0.98.125:34675)
0618 22:18:17.357520 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:17.357524 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:17.357858 (+ 334us) server_negotiation.cc:407] Connection header received
0618 22:18:17.358044 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:17.358047 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:17.358096 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:17.358176 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:17.358864 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.359669 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.360829 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.361191 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.361276 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:17.362845 (+ 1569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:17.362849 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:17.362852 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:17.362875 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:17.364289 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:17.364716 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:17.364720 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:17.364722 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:17.364808 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:17.365083 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:17.365092 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:17.365094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:17.365264 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:17.365368 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:17.365625 (+ 257us) server_negotiation.cc:299] Negotiation successful
0618 22:18:17.365742 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260618 22:18:17.365885 7256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:17.357482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:34675 (local address 127.0.0.1:54054)
0618 22:18:17.357774 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:17.357788 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:17.357948 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:17.358208 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:17.358210 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:17.358446 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:17.358733 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:17.358746 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.359791 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.359794 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:17.360708 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:17.360715 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.361435 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.361438 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:17.361662 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:17.362270 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:17.362284 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:17.362697 (+ 413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:18:17.364396 (+ 1699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:17.364403 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:17.364405 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:17.364618 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:17.364903 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:17.364906 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:17.364908 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:17.364991 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:17.365357 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:18:17.365359 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:18:17.365453 (+ 94us) client_negotiation.cc:769] Sending connection context
0618 22:18:17.365563 (+ 110us) client_negotiation.cc:240] Negotiation successful
0618 22:18:17.365703 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"mutex_wait_us":4,"thread_start_us":98,"threads_started":1}
I20260618 22:18:17.368680 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 6576
I20260618 22:18:17.376312 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:34925
--local_ip_for_outbound_sockets=127.0.98.65
--tserver_master_addrs=127.0.98.126:42267
--webserver_port=36469
--webserver_interface=127.0.98.65
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/krb5kdc/oryx_127.0.98.65.keytab
--principal=oryx/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:34497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.126:42267,127.0.98.125:34675
--tserver_master_addrs=127.0.98.126:42267,127.0.98.125:34675,127.0.98.124:36495 with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 22:18:17.387526 6553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:17.358023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59744 (local address 127.0.98.126:42267)
0618 22:18:17.358191 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:17.358195 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:17.358207 (+ 12us) server_negotiation.cc:407] Connection header received
0618 22:18:17.358243 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:17.358246 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:17.358295 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:17.358380 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:17.358920 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.359709 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.361368 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.361693 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.361773 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:17.383503 (+ 21730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:17.383507 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:17.383510 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:17.383541 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:17.385650 (+ 2109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:17.386209 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:17.386212 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:17.386214 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:17.386292 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:17.386587 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:17.386590 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:17.386592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:17.386808 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:17.386902 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:17.387235 (+ 333us) server_negotiation.cc:299] Negotiation successful
0618 22:18:17.387360 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260618 22:18:17.387777 7258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:17.357761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:36495 (local address 127.0.0.1:36426)
0618 22:18:17.358684 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:17.358698 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:17.358804 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:17.359070 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:17.359073 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:17.359253 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:17.359349 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:17.359355 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.360230 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.360232 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:17.361048 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:17.361056 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.361570 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.361572 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:17.361685 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:17.382580 (+ 20895us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:17.382608 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:17.383353 (+ 745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:18:17.385658 (+ 2305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:17.385664 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:17.385668 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:17.386003 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:17.386350 (+ 347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:17.386356 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:17.386361 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:17.386461 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:17.387054 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:18:17.387059 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:18:17.387199 (+ 140us) client_negotiation.cc:769] Sending connection context
0618 22:18:17.387315 (+ 116us) client_negotiation.cc:240] Negotiation successful
0618 22:18:17.387496 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":860,"thread_start_us":84,"threads_started":1}
I20260618 22:18:17.388266 7257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:17.357325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:42267 (local address 127.0.0.1:59744)
0618 22:18:17.357730 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:17.357744 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:17.357841 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:17.358422 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:17.358425 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:17.358643 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:17.358763 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:17.358770 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.359859 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.359862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:17.361232 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:17.361243 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.361957 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.361961 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:17.362059 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:17.382680 (+ 20621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:17.382696 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:17.383293 (+ 597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:18:17.385790 (+ 2497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:17.385793 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:17.385795 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:17.386094 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:17.386397 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:17.386400 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:17.386402 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:17.386492 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:17.386974 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:18:17.386977 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:18:17.387064 (+ 87us) client_negotiation.cc:769] Sending connection context
0618 22:18:17.387719 (+ 655us) client_negotiation.cc:240] Negotiation successful
0618 22:18:17.387870 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":100,"threads_started":1}
I20260618 22:18:17.388623 7235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:17.358505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36426 (local address 127.0.98.124:36495)
0618 22:18:17.358646 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:17.358650 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:17.358756 (+ 106us) server_negotiation.cc:407] Connection header received
0618 22:18:17.358888 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:17.358891 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:17.358923 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:17.358963 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:17.359451 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.360124 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.361155 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.361439 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.361504 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:17.383547 (+ 22043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:17.383554 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:17.383559 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:17.383590 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:17.385518 (+ 1928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:17.386128 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:17.386134 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:17.386137 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:17.386237 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:17.386570 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:17.386575 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:17.386579 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:17.386897 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:17.387012 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:17.388278 (+ 1266us) server_negotiation.cc:299] Negotiation successful
0618 22:18:17.388405 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":16,"server-negotiator.queue_time_us":28}
Jun 18 22:18:17 dist-test-slave-g6r4 krb5kdc[6469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821095, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.126@KRBTEST.COM for oryx/127.0.98.124@KRBTEST.COM
I20260618 22:18:17.401832 7071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:17.392259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:36495 (local address 127.0.0.1:36440)
0618 22:18:17.392420 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:17.392433 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:17.392512 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:17.392795 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:17.392798 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:17.392979 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:17.393076 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:17.393082 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.394200 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.394203 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:17.395175 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:17.395186 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.395876 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.395880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:17.395979 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:17.396478 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:17.396492 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:17.398082 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:18:17.400012 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:17.400016 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:17.400020 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:17.400296 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:17.400671 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:17.400676 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:17.400678 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:17.400779 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:17.401308 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:18:17.401312 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:18:17.401411 (+ 99us) client_negotiation.cc:769] Sending connection context
0618 22:18:17.401531 (+ 120us) client_negotiation.cc:240] Negotiation successful
0618 22:18:17.401662 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96}
I20260618 22:18:17.402002 7235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:17.392332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36440 (local address 127.0.98.124:36495)
0618 22:18:17.392436 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:17.392440 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:17.392475 (+ 35us) server_negotiation.cc:407] Connection header received
0618 22:18:17.392630 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:17.392633 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:17.392685 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:17.392770 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:17.393191 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.394064 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.395304 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.395714 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.395855 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:17.398219 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:17.398223 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:17.398226 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:17.398249 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:17.399897 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:17.400413 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:17.400416 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:17.400418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:17.400500 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:17.400953 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:17.400957 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:17.400959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:17.401204 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:17.401306 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:17.401653 (+ 347us) server_negotiation.cc:299] Negotiation successful
0618 22:18:17.401766 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260618 22:18:17.403263 6514 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.124:36495: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "9d8bc0e14290456db37208ee7d35b1cc"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.124"
port: 36495
}
attrs {
promote: true
}
}
dest_uuid: "cafe178b94854b17acaae5da06c1920d"
cas_config_opid_index: 5
I20260618 22:18:17.403906 6514 consensus_queue.cc:237] T 00000000000000000000000000000000 P cafe178b94854b17acaae5da06c1920d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "cafe178b94854b17acaae5da06c1920d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42267 } } peers { permanent_uuid: "512c63ff46bd41d6a51ca7e026cc8d47" member_type: NON_VOTER last_known_addr { host: "127.0.98.125" port: 34675 } attrs { promote: true } } peers { permanent_uuid: "9d8bc0e14290456db37208ee7d35b1cc" member_type: NON_VOTER last_known_addr { host: "127.0.98.124" port: 36495 } attrs { promote: true } }
I20260618 22:18:17.404929 7075 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cafe178b94854b17acaae5da06c1920d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 9d8bc0e14290456db37208ee7d35b1cc (127.0.98.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "cafe178b94854b17acaae5da06c1920d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42267 } } peers { permanent_uuid: "512c63ff46bd41d6a51ca7e026cc8d47" member_type: NON_VOTER last_known_addr { host: "127.0.98.125" port: 34675 } attrs { promote: true } } peers { permanent_uuid: "9d8bc0e14290456db37208ee7d35b1cc" member_type: NON_VOTER last_known_addr { host: "127.0.98.124" port: 36495 } attrs { promote: true } } }
I20260618 22:18:17.405884 7207 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:18:17.406807 7075 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.124:36495
I20260618 22:18:17.406987 7075 sys_catalog.cc:455] T 00000000000000000000000000000000 P cafe178b94854b17acaae5da06c1920d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "cafe178b94854b17acaae5da06c1920d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "cafe178b94854b17acaae5da06c1920d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42267 } } peers { permanent_uuid: "512c63ff46bd41d6a51ca7e026cc8d47" member_type: NON_VOTER last_known_addr { host: "127.0.98.125" port: 34675 } attrs { promote: true } } peers { permanent_uuid: "9d8bc0e14290456db37208ee7d35b1cc" member_type: NON_VOTER last_known_addr { host: "127.0.98.124" port: 36495 } attrs { promote: true } } }
I20260618 22:18:17.407073 7075 sys_catalog.cc:458] T 00000000000000000000000000000000 P cafe178b94854b17acaae5da06c1920d [sys.catalog]: This master's current role is: LEADER
I20260618 22:18:17.407343 7207 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc [term 1 FOLLOWER]: Refusing update from remote peer cafe178b94854b17acaae5da06c1920d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260618 22:18:17.407912 7074 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cafe178b94854b17acaae5da06c1920d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d8bc0e14290456db37208ee7d35b1cc" member_type: NON_VOTER last_known_addr { host: "127.0.98.124" port: 36495 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:18:17.408987 7087 master.cc:537] Master@127.0.98.124:36495 shutting down...
I20260618 22:18:17.412668 7207 raft_consensus.cc:858] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cafe178b94854b17acaae5da06c1920d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42267 } } peers { permanent_uuid: "512c63ff46bd41d6a51ca7e026cc8d47" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 34675 } } peers { permanent_uuid: "9d8bc0e14290456db37208ee7d35b1cc" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 36495 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "cafe178b94854b17acaae5da06c1920d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42267 } } peers { permanent_uuid: "512c63ff46bd41d6a51ca7e026cc8d47" member_type: NON_VOTER last_known_addr { host: "127.0.98.125" port: 34675 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "cafe178b94854b17acaae5da06c1920d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42267 } } peers { permanent_uuid: "512c63ff46bd41d6a51ca7e026cc8d47" member_type: NON_VOTER last_known_addr { host: "127.0.98.125" port: 34675 } attrs { promote: true } } peers { permanent_uuid: "9d8bc0e14290456db37208ee7d35b1cc" member_type: NON_VOTER last_known_addr { host: "127.0.98.124" port: 36495 } attrs { promote: true } }
W20260618 22:18:17.412787 7207 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cafe178b94854b17acaae5da06c1920d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42267 } } peers { permanent_uuid: "512c63ff46bd41d6a51ca7e026cc8d47" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 34675 } } peers { permanent_uuid: "9d8bc0e14290456db37208ee7d35b1cc" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 36495 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "cafe178b94854b17acaae5da06c1920d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42267 } } peers { permanent_uuid: "512c63ff46bd41d6a51ca7e026cc8d47" member_type: NON_VOTER last_known_addr { host: "127.0.98.125" port: 34675 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "cafe178b94854b17acaae5da06c1920d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42267 } } peers { permanent_uuid: "512c63ff46bd41d6a51ca7e026cc8d47" member_type: NON_VOTER last_known_addr { host: "127.0.98.125" port: 34675 } attrs { promote: true } } peers { permanent_uuid: "9d8bc0e14290456db37208ee7d35b1cc" member_type: NON_VOTER last_known_addr { host: "127.0.98.124" port: 36495 } attrs { promote: true } }
I20260618 22:18:17.414255 7207 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 512c63ff46bd41d6a51ca7e026cc8d47 (127.0.98.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "cafe178b94854b17acaae5da06c1920d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42267 } } peers { permanent_uuid: "512c63ff46bd41d6a51ca7e026cc8d47" member_type: NON_VOTER last_known_addr { host: "127.0.98.125" port: 34675 } attrs { promote: true } } }
I20260618 22:18:17.415630 7237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:17.405825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54068 (local address 127.0.98.125:34675)
0618 22:18:17.405966 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:17.405971 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:17.407681 (+ 1710us) server_negotiation.cc:407] Connection header received
0618 22:18:17.407871 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:17.407875 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:17.407933 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:17.407997 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:18:17.408443 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.409222 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.414117 (+ 4895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.414781 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.414871 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:17.415007 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:17.415431 (+ 424us) server_negotiation.cc:299] Negotiation successful
0618 22:18:17.415507 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260618 22:18:17.416128 7071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:17.405107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:34675 (local address 127.0.0.1:54068)
0618 22:18:17.407615 (+ 2508us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:17.407629 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:17.407743 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:17.408215 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:17.408218 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:17.408234 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:18:17.408318 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:17.408324 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.412506 (+ 4182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.412510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:17.413975 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:17.413987 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.415157 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.415161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:17.415268 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:17.415311 (+ 43us) client_negotiation.cc:769] Sending connection context
0618 22:18:17.415938 (+ 627us) client_negotiation.cc:240] Negotiation successful
0618 22:18:17.416013 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2423}
I20260618 22:18:17.416428 7239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader cafe178b94854b17acaae5da06c1920d. Latest consensus state: current_term: 1 leader_uuid: "cafe178b94854b17acaae5da06c1920d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "cafe178b94854b17acaae5da06c1920d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42267 } } peers { permanent_uuid: "512c63ff46bd41d6a51ca7e026cc8d47" member_type: NON_VOTER last_known_addr { host: "127.0.98.125" port: 34675 } attrs { promote: true } } }
I20260618 22:18:17.416545 7239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260618 22:18:17.416656 7239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cafe178b94854b17acaae5da06c1920d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "cafe178b94854b17acaae5da06c1920d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42267 } } peers { permanent_uuid: "512c63ff46bd41d6a51ca7e026cc8d47" member_type: NON_VOTER last_known_addr { host: "127.0.98.125" port: 34675 } attrs { promote: true } } }
I20260618 22:18:17.416694 7125 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 512c63ff46bd41d6a51ca7e026cc8d47 [term 1 LEARNER]: Refusing update from remote peer cafe178b94854b17acaae5da06c1920d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260618 22:18:17.416728 7239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260618 22:18:17.416807 7239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cafe178b94854b17acaae5da06c1920d" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "cafe178b94854b17acaae5da06c1920d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42267 } } peers { permanent_uuid: "512c63ff46bd41d6a51ca7e026cc8d47" member_type: NON_VOTER last_known_addr { host: "127.0.98.125" port: 34675 } attrs { promote: true } } }
I20260618 22:18:17.416903 7239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260618 22:18:17.417120 7074 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cafe178b94854b17acaae5da06c1920d [LEADER]: Connected to new peer: Peer: permanent_uuid: "512c63ff46bd41d6a51ca7e026cc8d47" member_type: NON_VOTER last_known_addr { host: "127.0.98.125" port: 34675 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260618 22:18:17.418979 7125 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 512c63ff46bd41d6a51ca7e026cc8d47 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 9d8bc0e14290456db37208ee7d35b1cc (127.0.98.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "cafe178b94854b17acaae5da06c1920d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42267 } } peers { permanent_uuid: "512c63ff46bd41d6a51ca7e026cc8d47" member_type: NON_VOTER last_known_addr { host: "127.0.98.125" port: 34675 } attrs { promote: true } } peers { permanent_uuid: "9d8bc0e14290456db37208ee7d35b1cc" member_type: NON_VOTER last_known_addr { host: "127.0.98.124" port: 36495 } attrs { promote: true } } }
I20260618 22:18:17.420789 7154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 512c63ff46bd41d6a51ca7e026cc8d47 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cafe178b94854b17acaae5da06c1920d. Latest consensus state: current_term: 1 leader_uuid: "cafe178b94854b17acaae5da06c1920d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "cafe178b94854b17acaae5da06c1920d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42267 } } peers { permanent_uuid: "512c63ff46bd41d6a51ca7e026cc8d47" member_type: NON_VOTER last_known_addr { host: "127.0.98.125" port: 34675 } attrs { promote: true } } peers { permanent_uuid: "9d8bc0e14290456db37208ee7d35b1cc" member_type: NON_VOTER last_known_addr { host: "127.0.98.124" port: 36495 } attrs { promote: true } } }
I20260618 22:18:17.420930 7154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 512c63ff46bd41d6a51ca7e026cc8d47 [sys.catalog]: This master's current role is: LEARNER
I20260618 22:18:17.421015 7154 sys_catalog.cc:455] T 00000000000000000000000000000000 P 512c63ff46bd41d6a51ca7e026cc8d47 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cafe178b94854b17acaae5da06c1920d" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "cafe178b94854b17acaae5da06c1920d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42267 } } peers { permanent_uuid: "512c63ff46bd41d6a51ca7e026cc8d47" member_type: NON_VOTER last_known_addr { host: "127.0.98.125" port: 34675 } attrs { promote: true } } peers { permanent_uuid: "9d8bc0e14290456db37208ee7d35b1cc" member_type: NON_VOTER last_known_addr { host: "127.0.98.124" port: 36495 } attrs { promote: true } } }
I20260618 22:18:17.421085 7154 sys_catalog.cc:458] T 00000000000000000000000000000000 P 512c63ff46bd41d6a51ca7e026cc8d47 [sys.catalog]: This master's current role is: LEARNER
I20260618 22:18:17.421677 7074 raft_consensus.cc:1050] T 00000000000000000000000000000000 P cafe178b94854b17acaae5da06c1920d: attempting to promote NON_VOTER 512c63ff46bd41d6a51ca7e026cc8d47 to VOTER
I20260618 22:18:17.422169 7074 consensus_queue.cc:237] T 00000000000000000000000000000000 P cafe178b94854b17acaae5da06c1920d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cafe178b94854b17acaae5da06c1920d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42267 } } peers { permanent_uuid: "512c63ff46bd41d6a51ca7e026cc8d47" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 34675 } attrs { promote: false } } peers { permanent_uuid: "9d8bc0e14290456db37208ee7d35b1cc" member_type: NON_VOTER last_known_addr { host: "127.0.98.124" port: 36495 } attrs { promote: true } }
W20260618 22:18:17.424656 6500 consensus_peers.cc:487] T 00000000000000000000000000000000 P cafe178b94854b17acaae5da06c1920d -> Peer 9d8bc0e14290456db37208ee7d35b1cc (127.0.98.124:36495): Couldn't send request to peer 9d8bc0e14290456db37208ee7d35b1cc. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260618 22:18:17.424661 7260 mvcc.cc:205] Tried to move back new op lower bound from 7298339209636028416 to 7298339209500454912. Current Snapshot: MvccSnapshot[applied={T|T < 7298339209536790528}]
I20260618 22:18:17.424988 7125 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 512c63ff46bd41d6a51ca7e026cc8d47 [term 1 LEARNER]: Refusing update from remote peer cafe178b94854b17acaae5da06c1920d: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260618 22:18:17.425395 7073 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cafe178b94854b17acaae5da06c1920d [LEADER]: Connected to new peer: Peer: permanent_uuid: "512c63ff46bd41d6a51ca7e026cc8d47" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 34675 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260618 22:18:17.426401 7087 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260618 22:18:17.426614 7087 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260618 22:18:17.426689 7087 tablet_replica.cc:330] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc: stopping tablet replica
I20260618 22:18:17.427599 7074 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cafe178b94854b17acaae5da06c1920d [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 512c63ff46bd41d6a51ca7e026cc8d47 (127.0.98.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cafe178b94854b17acaae5da06c1920d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42267 } } peers { permanent_uuid: "512c63ff46bd41d6a51ca7e026cc8d47" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 34675 } attrs { promote: false } } peers { permanent_uuid: "9d8bc0e14290456db37208ee7d35b1cc" member_type: NON_VOTER last_known_addr { host: "127.0.98.124" port: 36495 } attrs { promote: true } } }
I20260618 22:18:17.428198 7125 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 512c63ff46bd41d6a51ca7e026cc8d47 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 512c63ff46bd41d6a51ca7e026cc8d47 (127.0.98.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cafe178b94854b17acaae5da06c1920d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42267 } } peers { permanent_uuid: "512c63ff46bd41d6a51ca7e026cc8d47" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 34675 } attrs { promote: false } } peers { permanent_uuid: "9d8bc0e14290456db37208ee7d35b1cc" member_type: NON_VOTER last_known_addr { host: "127.0.98.124" port: 36495 } attrs { promote: true } } }
I20260618 22:18:17.429031 7074 sys_catalog.cc:455] T 00000000000000000000000000000000 P cafe178b94854b17acaae5da06c1920d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "cafe178b94854b17acaae5da06c1920d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cafe178b94854b17acaae5da06c1920d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42267 } } peers { permanent_uuid: "512c63ff46bd41d6a51ca7e026cc8d47" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 34675 } attrs { promote: false } } peers { permanent_uuid: "9d8bc0e14290456db37208ee7d35b1cc" member_type: NON_VOTER last_known_addr { host: "127.0.98.124" port: 36495 } attrs { promote: true } } }
I20260618 22:18:17.429132 7074 sys_catalog.cc:458] T 00000000000000000000000000000000 P cafe178b94854b17acaae5da06c1920d [sys.catalog]: This master's current role is: LEADER
I20260618 22:18:17.429206 7075 sys_catalog.cc:455] T 00000000000000000000000000000000 P cafe178b94854b17acaae5da06c1920d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cafe178b94854b17acaae5da06c1920d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cafe178b94854b17acaae5da06c1920d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42267 } } peers { permanent_uuid: "512c63ff46bd41d6a51ca7e026cc8d47" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 34675 } attrs { promote: false } } peers { permanent_uuid: "9d8bc0e14290456db37208ee7d35b1cc" member_type: NON_VOTER last_known_addr { host: "127.0.98.124" port: 36495 } attrs { promote: true } } }
I20260618 22:18:17.429317 7075 sys_catalog.cc:458] T 00000000000000000000000000000000 P cafe178b94854b17acaae5da06c1920d [sys.catalog]: This master's current role is: LEADER
I20260618 22:18:17.430716 7265 sys_catalog.cc:455] T 00000000000000000000000000000000 P 512c63ff46bd41d6a51ca7e026cc8d47 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cafe178b94854b17acaae5da06c1920d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cafe178b94854b17acaae5da06c1920d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42267 } } peers { permanent_uuid: "512c63ff46bd41d6a51ca7e026cc8d47" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 34675 } attrs { promote: false } } peers { permanent_uuid: "9d8bc0e14290456db37208ee7d35b1cc" member_type: NON_VOTER last_known_addr { host: "127.0.98.124" port: 36495 } attrs { promote: true } } }
I20260618 22:18:17.431017 7265 sys_catalog.cc:458] T 00000000000000000000000000000000 P 512c63ff46bd41d6a51ca7e026cc8d47 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:18:17.447368 7087 master.cc:552] Master@127.0.98.124:36495 shutdown complete.
I20260618 22:18:17.447476 7087 master_runner.cc:305] Clearing existing system tablet
I20260618 22:18:17.447737 7087 fs_manager.cc:359] Metadata directory not provided
I20260618 22:18:17.447798 7087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:18:17.449177 7087 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260618 22:18:17.450574 7270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:17.450811 7087 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:18:17.450937 7087 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 22:18:17.450968 7087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/master-2/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/master-2/wal
uuid: "9d8bc0e14290456db37208ee7d35b1cc"
format_stamp: "Formatted at 2026-06-18 22:18:17 on dist-test-slave-g6r4"
server_key: "e2b6e620a7cd5c75a5639ab2b59b4df7"
server_key_iv: "009abac25c6c03286a2d189fa1c8c7af"
server_key_version: "encryptionkey@0"
I20260618 22:18:17.452598 7087 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:18:17.454830 7087 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260618 22:18:17.454952 7087 log.cc:1196] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc: Deleting WAL directory at /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260618 22:18:17.455273 7087 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc: Deleting consensus metadata
I20260618 22:18:17.455426 7087 master_runner.cc:315] Copying system tablet from 127.0.98.126:42267
I20260618 22:18:17.456534 7087 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.126:42267
I20260618 22:18:17.465476 6553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:17.457158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59748 (local address 127.0.98.126:42267)
0618 22:18:17.457303 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:17.457307 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:17.457562 (+ 255us) server_negotiation.cc:407] Connection header received
0618 22:18:17.457673 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:17.457676 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:17.457731 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:17.457816 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:17.458327 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.459112 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.460265 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.460605 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.460711 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:17.461993 (+ 1282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:17.461998 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:17.462004 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:17.462030 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:17.463741 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:17.464220 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:17.464224 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:17.464226 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:17.464311 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:17.464601 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:17.464604 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:17.464606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:17.464812 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:17.464918 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:17.465204 (+ 286us) server_negotiation.cc:299] Negotiation successful
0618 22:18:17.465321 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260618 22:18:17.465471 7276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:17.457158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:42267 (local address 127.0.0.1:59748)
0618 22:18:17.457450 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:17.457466 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:17.457586 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:17.457850 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:17.457853 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:17.458092 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:17.458195 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:17.458202 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.459234 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.459237 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:17.460144 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:17.460153 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.460723 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.460726 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:17.460817 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:17.461384 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:17.461402 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:17.461816 (+ 414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:18:17.463858 (+ 2042us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:17.463862 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:17.463864 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:17.464116 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:17.464408 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:17.464411 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:17.464413 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:17.464504 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:17.464920 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:18:17.464924 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:18:17.465019 (+ 95us) client_negotiation.cc:769] Sending connection context
0618 22:18:17.465130 (+ 111us) client_negotiation.cc:240] Negotiation successful
0618 22:18:17.465266 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":87,"threads_started":1}
I20260618 22:18:17.466336 6534 tablet_copy_service.cc:140] P cafe178b94854b17acaae5da06c1920d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 9d8bc0e14290456db37208ee7d35b1cc ({username='oryx', principal='oryx/127.0.98.124@KRBTEST.COM'} at 127.0.0.1:59748)
I20260618 22:18:17.466434 6534 tablet_copy_service.cc:161] P cafe178b94854b17acaae5da06c1920d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 9d8bc0e14290456db37208ee7d35b1cc at {username='oryx', principal='oryx/127.0.98.124@KRBTEST.COM'} at 127.0.0.1:59748: session id = 9d8bc0e14290456db37208ee7d35b1cc-00000000000000000000000000000000
I20260618 22:18:17.467480 6534 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P cafe178b94854b17acaae5da06c1920d: Tablet Copy: opened 0 blocks and 1 log segments
I20260618 22:18:17.468644 7087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260618 22:18:17.471102 7087 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc: tablet copy: Starting download of 0 data blocks...
I20260618 22:18:17.471283 7087 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc: tablet copy: Starting download of 1 WAL segments...
I20260618 22:18:17.472738 7087 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260618 22:18:17.474422 6534 tablet_copy_service.cc:342] P cafe178b94854b17acaae5da06c1920d: Request end of tablet copy session 9d8bc0e14290456db37208ee7d35b1cc-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.124@KRBTEST.COM'} at 127.0.0.1:59748
I20260618 22:18:17.474510 6534 tablet_copy_service.cc:433] P cafe178b94854b17acaae5da06c1920d: ending tablet copy session 9d8bc0e14290456db37208ee7d35b1cc-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 9d8bc0e14290456db37208ee7d35b1cc
I20260618 22:18:17.481153 7087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:18:17.482865 7277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:17.483011 7280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:17.483628 7087 server_base.cc:1034] running on GCE node
W20260618 22:18:17.486850 7278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:17.487185 7087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:18:17.487493 7087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:18:17.488641 7087 hybrid_clock.cc:648] HybridClock initialized: now 1781821097488627 us; error 32 us; skew 500 ppm
Jun 18 22:18:17 dist-test-slave-g6r4 krb5kdc[6469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821097, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:18:17.490362 7087 init.cc:377] Logged in from keytab as oryx/127.0.98.124@KRBTEST.COM (short username oryx)
I20260618 22:18:17.490792 7087 webserver.cc:460] Webserver started at http://127.0.98.124:35999/ using document root <none> and password file <none>
I20260618 22:18:17.490974 7087 fs_manager.cc:359] Metadata directory not provided
I20260618 22:18:17.491016 7087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:18:17.492303 7087 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260618 22:18:17.493350 7287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:17.493598 7087 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 22:18:17.493710 7087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/master-2/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/master-2/wal
uuid: "9d8bc0e14290456db37208ee7d35b1cc"
format_stamp: "Formatted at 2026-06-18 22:18:17 on dist-test-slave-g6r4"
server_key: "e2b6e620a7cd5c75a5639ab2b59b4df7"
server_key_iv: "009abac25c6c03286a2d189fa1c8c7af"
server_key_version: "encryptionkey@0"
I20260618 22:18:17.493846 7087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:18:17.509986 7087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:18:17.512118 7087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:18:17.516861 7087 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.124:36495
I20260618 22:18:17.517444 7087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/master-2/data/info.pb
I20260618 22:18:17.518342 7339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.124:36495 every 8 connection(s)
I20260618 22:18:17.520493 7340 sys_catalog.cc:263] Verifying existing consensus state
I20260618 22:18:17.521497 7340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc: Bootstrap starting.
W20260618 22:18:17.527612 7259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:18:17.527858 7259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:18:17.527935 7259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260618 22:18:17.529320 7340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260618 22:18:17.529579 7340 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc: Bootstrap complete.
I20260618 22:18:17.530066 7340 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cafe178b94854b17acaae5da06c1920d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42267 } } peers { permanent_uuid: "512c63ff46bd41d6a51ca7e026cc8d47" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 34675 } attrs { promote: false } } peers { permanent_uuid: "9d8bc0e14290456db37208ee7d35b1cc" member_type: NON_VOTER last_known_addr { host: "127.0.98.124" port: 36495 } attrs { promote: true } }
I20260618 22:18:17.530191 7340 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9d8bc0e14290456db37208ee7d35b1cc, State: Initialized, Role: LEARNER
I20260618 22:18:17.530288 7340 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cafe178b94854b17acaae5da06c1920d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42267 } } peers { permanent_uuid: "512c63ff46bd41d6a51ca7e026cc8d47" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 34675 } attrs { promote: false } } peers { permanent_uuid: "9d8bc0e14290456db37208ee7d35b1cc" member_type: NON_VOTER last_known_addr { host: "127.0.98.124" port: 36495 } attrs { promote: true } }
I20260618 22:18:17.530583 7346 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cafe178b94854b17acaae5da06c1920d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42267 } } peers { permanent_uuid: "512c63ff46bd41d6a51ca7e026cc8d47" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 34675 } attrs { promote: false } } peers { permanent_uuid: "9d8bc0e14290456db37208ee7d35b1cc" member_type: NON_VOTER last_known_addr { host: "127.0.98.124" port: 36495 } attrs { promote: true } } }
I20260618 22:18:17.530684 7340 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:18:17.530857 7346 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc [sys.catalog]: This master's current role is: LEARNER
W20260618 22:18:17.533082 7259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:18:17.533140 7259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:18:17.533260 7259 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260618 22:18:17.534010 7357 catalog_manager.cc:1259] Loaded cluster ID: fc2c6ba654e446178226c06313f3379d
I20260618 22:18:17.534062 7357 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc: loading cluster ID for follower catalog manager: success
I20260618 22:18:17.534428 6839 heartbeater.cc:503] Master 127.0.98.126:42267 was elected leader, sending a full tablet report...
I20260618 22:18:17.536736 7357 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc: acquiring CA information for follower catalog manager: success
I20260618 22:18:17.538036 7357 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260618 22:18:17.539812 7259 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:34497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/krb5kdc/oryx_127.0.98.65.keytab
--principal=oryx/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:34925
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=36469
--tserver_master_addrs=127.0.98.126:42267,127.0.98.125:34675,127.0.98.124:36495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7259
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:18:17.540704 7259 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:18:17.541576 7259 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:18:17.550541 7359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:17.550491 7360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:17.550607 7259 server_base.cc:1034] running on GCE node
W20260618 22:18:17.550503 7362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:17.551237 7259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:18:17.551841 7259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:18:17.552994 7259 hybrid_clock.cc:648] HybridClock initialized: now 1781821097552980 us; error 26 us; skew 500 ppm
Jun 18 22:18:17 dist-test-slave-g6r4 krb5kdc[6469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821097, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:18:17.555570 7259 init.cc:377] Logged in from keytab as oryx/127.0.98.65@KRBTEST.COM (short username oryx)
I20260618 22:18:17.556340 7259 webserver.cc:460] Webserver started at http://127.0.98.65:36469/ using document root <none> and password file <none>
I20260618 22:18:17.556813 7259 fs_manager.cc:359] Metadata directory not provided
I20260618 22:18:17.556874 7259 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:18:17.560285 7259 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260618 22:18:17.562521 7369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:17.563681 7259 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260618 22:18:17.563802 7259 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-0/wal
uuid: "4c35cb6b49cd461b83b7ea8f6e4dcb54"
format_stamp: "Formatted at 2026-06-18 22:18:16 on dist-test-slave-g6r4"
server_key: "83fb473682df8aef856c7fd6c526c6a4"
server_key_iv: "0f86e99ac01b87aed6ff72fe2ffcd7c8"
server_key_version: "encryptionkey@0"
I20260618 22:18:17.564272 7259 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:18:17.607802 7259 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:18:17.610845 7259 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:18:17.611044 7259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:18:17.611647 7259 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:18:17.612489 7259 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:18:17.612536 7259 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:17.612603 7259 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:18:17.612633 7259 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:17.623358 7259 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:34925
I20260618 22:18:17.623376 7482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:34925 every 8 connection(s)
I20260618 22:18:17.624420 7259 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
I20260618 22:18:17.629381 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 7259
I20260618 22:18:17.629563 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 6710
Jun 18 22:18:17 dist-test-slave-g6r4 krb5kdc[6469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821097, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM
Jun 18 22:18:17 dist-test-slave-g6r4 krb5kdc[6469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821097, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.124@KRBTEST.COM
Jun 18 22:18:17 dist-test-slave-g6r4 krb5kdc[6469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821097, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.125@KRBTEST.COM
I20260618 22:18:17.643409 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:41835
--local_ip_for_outbound_sockets=127.0.98.66
--tserver_master_addrs=127.0.98.126:42267
--webserver_port=46797
--webserver_interface=127.0.98.66
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/krb5kdc/oryx_127.0.98.66.keytab
--principal=oryx/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:34497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.126:42267,127.0.98.125:34675
--tserver_master_addrs=127.0.98.126:42267,127.0.98.125:34675,127.0.98.124:36495 with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 22:18:17.645536 6553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:17.627934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:58953 (local address 127.0.98.126:42267)
0618 22:18:17.628053 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:17.628056 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:17.628077 (+ 21us) server_negotiation.cc:407] Connection header received
0618 22:18:17.629340 (+ 1263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:17.629345 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:17.629403 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:17.629495 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:17.630845 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.631666 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.633801 (+ 2135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.634177 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.634266 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:17.640819 (+ 6553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:17.640826 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:17.640833 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:17.640867 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:17.643149 (+ 2282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:17.643726 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:17.643731 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:17.643734 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:17.643828 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:17.644234 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:17.644239 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:17.644241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:17.644481 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:17.644581 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:17.645234 (+ 653us) server_negotiation.cc:299] Negotiation successful
0618 22:18:17.645350 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260618 22:18:17.646126 7487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:17.626296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:50121 (local address 127.0.98.124:36495)
0618 22:18:17.626599 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:17.626614 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:17.627565 (+ 951us) server_negotiation.cc:407] Connection header received
0618 22:18:17.629043 (+ 1478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:17.629046 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:17.629098 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:17.629200 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:17.630840 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.632039 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.634956 (+ 2917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.635384 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.635470 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:17.640930 (+ 5460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:17.640937 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:17.640969 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:17.641007 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:17.644313 (+ 3306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:17.644780 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:17.644784 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:17.644785 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:17.644868 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:17.645162 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:17.645169 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:17.645173 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:17.645390 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:17.645471 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:17.645831 (+ 360us) server_negotiation.cc:299] Negotiation successful
0618 22:18:17.645978 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":84,"threads_started":1}
I20260618 22:18:17.647331 7237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:17.626286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:37443 (local address 127.0.98.125:34675)
0618 22:18:17.626417 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:17.626422 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:17.627412 (+ 990us) server_negotiation.cc:407] Connection header received
0618 22:18:17.628734 (+ 1322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:17.628738 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:17.628805 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:17.628910 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:17.630875 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.631717 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.635286 (+ 3569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.635705 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.635826 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:17.642299 (+ 6473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:17.642305 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:17.642310 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:17.642348 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:17.645228 (+ 2880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:17.645834 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:17.645838 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:17.645840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:17.645961 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:17.646335 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:17.646338 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:17.646340 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:17.646572 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:17.646669 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:17.647052 (+ 383us) server_negotiation.cc:299] Negotiation successful
0618 22:18:17.647170 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260618 22:18:17.647745 7490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:17.627714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:42267 (local address 127.0.98.65:58953)
0618 22:18:17.627980 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:17.627993 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:17.628433 (+ 440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:17.629628 (+ 1195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:17.629631 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:17.629866 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:17.630663 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:17.630683 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.631860 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.631865 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:17.633648 (+ 1783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:17.633661 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.634489 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.634494 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:17.634715 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:17.637717 (+ 3002us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:17.637740 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:17.640614 (+ 2874us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:18:17.643269 (+ 2655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:17.643274 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:17.643289 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:17.643604 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:17.643954 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:17.643958 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:17.643962 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:17.644133 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:17.644636 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:18:17.644643 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:18:17.644965 (+ 322us) client_negotiation.cc:769] Sending connection context
0618 22:18:17.645964 (+ 999us) client_negotiation.cc:240] Negotiation successful
0618 22:18:17.646619 (+ 655us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":365,"thread_start_us":102,"threads_started":1}
I20260618 22:18:17.648041 7488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:17.626696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:34675 (local address 127.0.98.65:37443)
0618 22:18:17.627263 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:17.627317 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:17.628433 (+ 1116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:17.629108 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:17.629119 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:17.630142 (+ 1023us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:17.630665 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:17.630683 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.632013 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.632018 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:17.633707 (+ 1689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:17.633718 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.635892 (+ 2174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.635897 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:17.636015 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:17.640516 (+ 4501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:17.640537 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:17.642100 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:18:17.645394 (+ 3294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:17.645398 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:17.645402 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:17.645697 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:17.646107 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:17.646110 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:17.646112 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:17.646214 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:17.646747 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:18:17.646751 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:18:17.646851 (+ 100us) client_negotiation.cc:769] Sending connection context
0618 22:18:17.646981 (+ 130us) client_negotiation.cc:240] Negotiation successful
0618 22:18:17.647136 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":316,"mutex_wait_us":229,"spinlock_wait_cycles":22912,"thread_start_us":582,"threads_started":1}
I20260618 22:18:17.648809 7489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:17.626696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:36495 (local address 127.0.98.65:50121)
0618 22:18:17.627478 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:17.627493 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:17.628433 (+ 940us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:17.629253 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:17.629257 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:17.630132 (+ 875us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:17.630666 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:17.630684 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.633325 (+ 2641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.633328 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:17.634804 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:17.634826 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.635638 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.635642 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:17.635749 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:17.637872 (+ 2123us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:17.637912 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:17.640695 (+ 2783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:18:17.644430 (+ 3735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:17.644433 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:17.644435 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:17.644673 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:17.644974 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:17.644977 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:17.644978 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:17.645054 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:17.645539 (+ 485us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:18:17.645543 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:18:17.645636 (+ 93us) client_negotiation.cc:769] Sending connection context
0618 22:18:17.646324 (+ 688us) client_negotiation.cc:240] Negotiation successful
0618 22:18:17.648526 (+ 2202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":703,"mutex_wait_us":624,"spinlock_wait_cycles":6912,"thread_start_us":159,"threads_started":1}
I20260618 22:18:17.649098 7485 heartbeater.cc:347] Connected to a master server at 127.0.98.125:34675
I20260618 22:18:17.649199 7484 heartbeater.cc:347] Connected to a master server at 127.0.98.126:42267
I20260618 22:18:17.649338 7485 heartbeater.cc:464] Registering TS with master...
I20260618 22:18:17.649353 7484 heartbeater.cc:464] Registering TS with master...
I20260618 22:18:17.649953 7483 heartbeater.cc:347] Connected to a master server at 127.0.98.124:36495
I20260618 22:18:17.650041 7485 heartbeater.cc:511] Master 127.0.98.125:34675 requested a full tablet report, sending...
I20260618 22:18:17.650041 7483 heartbeater.cc:464] Registering TS with master...
I20260618 22:18:17.650345 7484 heartbeater.cc:511] Master 127.0.98.126:42267 requested a full tablet report, sending...
I20260618 22:18:17.650398 7483 heartbeater.cc:511] Master 127.0.98.124:36495 requested a full tablet report, sending...
I20260618 22:18:17.651549 6514 ts_manager.cc:194] Re-registered known tserver with Master: 4c35cb6b49cd461b83b7ea8f6e4dcb54 (127.0.98.65:34925)
I20260618 22:18:17.651818 7295 ts_manager.cc:194] Registered new tserver with Master: 4c35cb6b49cd461b83b7ea8f6e4dcb54 (127.0.98.65:34925)
I20260618 22:18:17.651821 7115 ts_manager.cc:194] Registered new tserver with Master: 4c35cb6b49cd461b83b7ea8f6e4dcb54 (127.0.98.65:34925)
I20260618 22:18:17.652313 6514 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.65@KRBTEST.COM'} at 127.0.98.65:58953
I20260618 22:18:17.742690 6973 heartbeater.cc:503] Master 127.0.98.126:42267 was elected leader, sending a full tablet report...
W20260618 22:18:17.756225 7491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:18:17.756438 7491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:18:17.756503 7491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:18:17.760424 7491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:18:17.760473 7491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:18:17.760562 7491 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260618 22:18:17.765468 7491 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:34497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/krb5kdc/oryx_127.0.98.66.keytab
--principal=oryx/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:41835
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=46797
--tserver_master_addrs=127.0.98.126:42267,127.0.98.125:34675,127.0.98.124:36495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7491
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:18:17.766417 7491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:18:17.767373 7491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260618 22:18:17.777074 7491 server_base.cc:1034] running on GCE node
W20260618 22:18:17.777040 7497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:17.777040 7499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:17.777045 7496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:17.777648 7491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:18:17.778302 7491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:18:17.779490 7491 hybrid_clock.cc:648] HybridClock initialized: now 1781821097779470 us; error 31 us; skew 500 ppm
Jun 18 22:18:17 dist-test-slave-g6r4 krb5kdc[6469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821097, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:18:17.782308 7491 init.cc:377] Logged in from keytab as oryx/127.0.98.66@KRBTEST.COM (short username oryx)
I20260618 22:18:17.783033 7491 webserver.cc:460] Webserver started at http://127.0.98.66:46797/ using document root <none> and password file <none>
I20260618 22:18:17.783558 7491 fs_manager.cc:359] Metadata directory not provided
I20260618 22:18:17.783627 7491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:18:17.787088 7491 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:18:17.788906 7506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:17.790115 7491 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:18:17.790228 7491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-1/wal
uuid: "d7dc899699d84381a27dbdf8dae05cc2"
format_stamp: "Formatted at 2026-06-18 22:18:16 on dist-test-slave-g6r4"
server_key: "f763b4a3ec406bce7106d4dbc831362c"
server_key_iv: "e1102861e62f775def89b3d48caa7285"
server_key_version: "encryptionkey@0"
I20260618 22:18:17.790712 7491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:18:17.804584 7491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:18:17.807513 7491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:18:17.807698 7491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:18:17.808269 7491 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:18:17.809067 7491 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:18:17.809113 7491 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:17.809180 7491 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:18:17.809216 7491 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:17.817855 7491 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:41835
I20260618 22:18:17.817968 7619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:41835 every 8 connection(s)
I20260618 22:18:17.818894 7491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
I20260618 22:18:17.821796 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 7491
I20260618 22:18:17.821987 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 6844
I20260618 22:18:17.828305 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:46203
--local_ip_for_outbound_sockets=127.0.98.67
--tserver_master_addrs=127.0.98.126:42267
--webserver_port=40627
--webserver_interface=127.0.98.67
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/krb5kdc/oryx_127.0.98.67.keytab
--principal=oryx/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:34497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.126:42267,127.0.98.125:34675
--tserver_master_addrs=127.0.98.126:42267,127.0.98.125:34675,127.0.98.124:36495 with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 18 22:18:17 dist-test-slave-g6r4 krb5kdc[6469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821097, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM
Jun 18 22:18:17 dist-test-slave-g6r4 krb5kdc[6469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821097, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.125@KRBTEST.COM
Jun 18 22:18:17 dist-test-slave-g6r4 krb5kdc[6469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821097, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.124@KRBTEST.COM
I20260618 22:18:17.836457 7237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:17.820626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:50167 (local address 127.0.98.125:34675)
0618 22:18:17.820922 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:17.820928 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:17.824042 (+ 3114us) server_negotiation.cc:407] Connection header received
0618 22:18:17.824141 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:17.824144 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:17.824205 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:17.824287 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:17.825322 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.826185 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.829116 (+ 2931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.829490 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.829615 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:17.832238 (+ 2623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:17.832242 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:17.832246 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:17.832280 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:17.834029 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:17.834563 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:17.834566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:17.834568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:17.834770 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:17.835168 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:17.835171 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:17.835173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:17.835391 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:17.835457 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:17.836181 (+ 724us) server_negotiation.cc:299] Negotiation successful
0618 22:18:17.836301 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224}
I20260618 22:18:17.836457 6553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:17.820772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:39507 (local address 127.0.98.126:42267)
0618 22:18:17.820909 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:17.820914 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:17.821862 (+ 948us) server_negotiation.cc:407] Connection header received
0618 22:18:17.823268 (+ 1406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:17.823274 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:17.823338 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:17.823464 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:17.825403 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.826201 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.827526 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.827831 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.827922 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:17.831154 (+ 3232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:17.831158 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:17.831161 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:17.831188 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:17.833814 (+ 2626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:17.834560 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:17.834564 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:17.834566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:17.834659 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:17.835171 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:17.835174 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:17.835175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:17.835382 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:17.835465 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:17.836184 (+ 719us) server_negotiation.cc:299] Negotiation successful
0618 22:18:17.836301 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260618 22:18:17.836907 7487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:17.822561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:54807 (local address 127.0.98.124:36495)
0618 22:18:17.822692 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:17.822712 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:17.824335 (+ 1623us) server_negotiation.cc:407] Connection header received
0618 22:18:17.824518 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:17.824523 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:17.824585 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:17.824639 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:17.826456 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.827311 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.829722 (+ 2411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.830168 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.830253 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:17.833025 (+ 2772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:17.833031 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:17.833037 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:17.833073 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:17.834976 (+ 1903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:17.835439 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:17.835443 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:17.835445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:17.835532 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:17.835851 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:17.835854 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:17.835856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:17.836070 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:17.836137 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:17.836526 (+ 389us) server_negotiation.cc:299] Negotiation successful
0618 22:18:17.836666 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260618 22:18:17.838174 7625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:17.822243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:36495 (local address 127.0.98.66:54807)
0618 22:18:17.824255 (+ 2012us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:17.824269 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:17.824413 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:17.824793 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:17.824798 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:17.825025 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:17.826348 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:17.826364 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.827774 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.827780 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:17.829575 (+ 1795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:17.829587 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.830444 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.830450 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:17.830554 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:17.831112 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:17.831133 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:17.832850 (+ 1717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:18:17.835101 (+ 2251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:17.835104 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:17.835106 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:17.835323 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:17.835644 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:17.835647 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:17.835648 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:17.835733 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:17.836211 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:18:17.836213 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:18:17.836317 (+ 104us) client_negotiation.cc:769] Sending connection context
0618 22:18:17.837106 (+ 789us) client_negotiation.cc:240] Negotiation successful
0618 22:18:17.837280 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1929,"mutex_wait_us":1150,"thread_start_us":70,"threads_started":1}
I20260618 22:18:17.838174 7624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:17.821099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:42267 (local address 127.0.98.66:39507)
0618 22:18:17.821692 (+ 593us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:17.821749 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:17.823037 (+ 1288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:17.823775 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:17.823787 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:17.824318 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:17.825279 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:17.825289 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.826465 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.826467 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:17.827403 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:17.827412 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.827963 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.827967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:17.828179 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:17.828981 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:17.828997 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:17.830957 (+ 1960us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:18:17.833983 (+ 3026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:17.833991 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:17.834005 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:17.834442 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:17.834906 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:17.834909 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:17.834912 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:17.835064 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:17.835534 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:18:17.835539 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:18:17.835881 (+ 342us) client_negotiation.cc:769] Sending connection context
0618 22:18:17.836653 (+ 772us) client_negotiation.cc:240] Negotiation successful
0618 22:18:17.837051 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"mutex_wait_us":169,"spinlock_wait_cycles":18560,"thread_start_us":666,"threads_started":1}
I20260618 22:18:17.838944 7626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:17.821099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:34675 (local address 127.0.98.66:50167)
0618 22:18:17.823930 (+ 2831us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:17.823947 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:17.824052 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:17.824401 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:17.824404 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:17.824579 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:17.825156 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:17.825175 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.826346 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.826352 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:17.828982 (+ 2630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:17.828996 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.829773 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.829780 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:17.829888 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:17.830545 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:17.830560 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:17.832066 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:18:17.834138 (+ 2072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:17.834140 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:17.834142 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:17.834461 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:17.834878 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:17.834880 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:17.834882 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:17.835057 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:17.835533 (+ 476us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:18:17.835539 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:18:17.835881 (+ 342us) client_negotiation.cc:769] Sending connection context
0618 22:18:17.837513 (+ 1632us) client_negotiation.cc:240] Negotiation successful
0618 22:18:17.837645 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2723,"mutex_wait_us":110,"spinlock_wait_cycles":20864,"thread_start_us":170,"threads_started":1}
I20260618 22:18:17.839581 7621 heartbeater.cc:347] Connected to a master server at 127.0.98.126:42267
I20260618 22:18:17.839576 7620 heartbeater.cc:347] Connected to a master server at 127.0.98.124:36495
I20260618 22:18:17.839684 7622 heartbeater.cc:347] Connected to a master server at 127.0.98.125:34675
I20260618 22:18:17.839825 7622 heartbeater.cc:464] Registering TS with master...
I20260618 22:18:17.839828 7621 heartbeater.cc:464] Registering TS with master...
I20260618 22:18:17.839828 7620 heartbeater.cc:464] Registering TS with master...
I20260618 22:18:17.840332 7622 heartbeater.cc:511] Master 127.0.98.125:34675 requested a full tablet report, sending...
I20260618 22:18:17.840338 7621 heartbeater.cc:511] Master 127.0.98.126:42267 requested a full tablet report, sending...
I20260618 22:18:17.840350 7620 heartbeater.cc:511] Master 127.0.98.124:36495 requested a full tablet report, sending...
I20260618 22:18:17.841405 7295 ts_manager.cc:194] Registered new tserver with Master: d7dc899699d84381a27dbdf8dae05cc2 (127.0.98.66:41835)
I20260618 22:18:17.841404 7115 ts_manager.cc:194] Registered new tserver with Master: d7dc899699d84381a27dbdf8dae05cc2 (127.0.98.66:41835)
I20260618 22:18:17.841454 6514 ts_manager.cc:194] Re-registered known tserver with Master: d7dc899699d84381a27dbdf8dae05cc2 (127.0.98.66:41835)
I20260618 22:18:17.842343 6514 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.66@KRBTEST.COM'} at 127.0.98.66:39507
I20260618 22:18:17.920251 7071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:17.916366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:36495 (local address 127.0.0.1:36456)
0618 22:18:17.916582 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:17.916596 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:17.916691 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:17.917016 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:17.917018 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:17.917033 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:18:17.917117 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:17.917121 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.918166 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.918169 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:17.919085 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:17.919096 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.919957 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.919959 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:17.920023 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:17.920043 (+ 20us) client_negotiation.cc:769] Sending connection context
0618 22:18:17.920091 (+ 48us) client_negotiation.cc:240] Negotiation successful
0618 22:18:17.920145 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107}
I20260618 22:18:17.920311 7487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:17.916469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36456 (local address 127.0.98.124:36495)
0618 22:18:17.916604 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:17.916609 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:17.916652 (+ 43us) server_negotiation.cc:407] Connection header received
0618 22:18:17.916803 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:17.916806 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:17.916884 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:17.916997 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:18:17.917237 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.918050 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.919205 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:17.919840 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:17.919954 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:17.920086 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:17.920119 (+ 33us) server_negotiation.cc:299] Negotiation successful
0618 22:18:17.920182 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260618 22:18:17.920951 7314 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260618 22:18:17.921214 7346 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader cafe178b94854b17acaae5da06c1920d. Latest consensus state: current_term: 1 leader_uuid: "cafe178b94854b17acaae5da06c1920d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cafe178b94854b17acaae5da06c1920d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42267 } } peers { permanent_uuid: "512c63ff46bd41d6a51ca7e026cc8d47" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 34675 } attrs { promote: false } } peers { permanent_uuid: "9d8bc0e14290456db37208ee7d35b1cc" member_type: NON_VOTER last_known_addr { host: "127.0.98.124" port: 36495 } attrs { promote: true } } }
I20260618 22:18:17.921358 7346 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc [sys.catalog]: This master's current role is: LEARNER
W20260618 22:18:17.948479 7627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:18:17.948709 7627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:18:17.948781 7627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:18:17.952788 7627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:18:17.952844 7627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:18:17.952935 7627 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260618 22:18:17.958128 7627 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:34497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/krb5kdc/oryx_127.0.98.67.keytab
--principal=oryx/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:46203
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=40627
--tserver_master_addrs=127.0.98.126:42267,127.0.98.125:34675,127.0.98.124:36495
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:18:17.959115 7627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:18:17.960107 7627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:18:17.969659 7632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:17.969681 7627 server_base.cc:1034] running on GCE node
W20260618 22:18:17.969758 7633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:17.969641 7635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:17.970373 7627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:18:17.971139 7627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:18:17.972317 7627 hybrid_clock.cc:648] HybridClock initialized: now 1781821097972302 us; error 26 us; skew 500 ppm
Jun 18 22:18:17 dist-test-slave-g6r4 krb5kdc[6469](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821097, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:18:17.974998 7627 init.cc:377] Logged in from keytab as oryx/127.0.98.67@KRBTEST.COM (short username oryx)
I20260618 22:18:17.975844 7627 webserver.cc:460] Webserver started at http://127.0.98.67:40627/ using document root <none> and password file <none>
I20260618 22:18:17.976374 7627 fs_manager.cc:359] Metadata directory not provided
I20260618 22:18:17.976476 7627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:18:17.980086 7627 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:18:17.982092 7642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:17.983237 7627 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:18:17.983371 7627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-2/wal
uuid: "5256bda72de544e985aaad69508c7748"
format_stamp: "Formatted at 2026-06-18 22:18:16 on dist-test-slave-g6r4"
server_key: "0e3adbfbc258158b0970603c0bf1fc94"
server_key_iv: "9b5e649e3867a672fadfd64652f7ee60"
server_key_version: "encryptionkey@0"
I20260618 22:18:17.983880 7627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:18:17.995357 7627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:18:17.998507 7627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:18:17.998690 7627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:18:17.999197 7627 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:18:17.999971 7627 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:18:18.000041 7627 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:18.000106 7627 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:18:18.000155 7627 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:18.009099 7627 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:46203
I20260618 22:18:18.009128 7755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:46203 every 8 connection(s)
I20260618 22:18:18.010061 7627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
I20260618 22:18:18.014928 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 7627
Jun 18 22:18:18 dist-test-slave-g6r4 krb5kdc[6469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821097, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.124@KRBTEST.COM
Jun 18 22:18:18 dist-test-slave-g6r4 krb5kdc[6469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821097, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.125@KRBTEST.COM
Jun 18 22:18:18 dist-test-slave-g6r4 krb5kdc[6469](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821097, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM
I20260618 22:18:18.027386 7237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:18.012192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:59315 (local address 127.0.98.125:34675)
0618 22:18:18.012341 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:18.012348 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:18.013317 (+ 969us) server_negotiation.cc:407] Connection header received
0618 22:18:18.014076 (+ 759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:18.014080 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:18.014134 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:18.014225 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:18.015514 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:18.016324 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:18.018880 (+ 2556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:18.019216 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:18.019311 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:18.023293 (+ 3982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:18.023300 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:18.023306 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:18.023349 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:18.025508 (+ 2159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:18.026040 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:18.026044 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:18.026045 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:18.026136 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:18.026454 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:18.026459 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:18.026461 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:18.026684 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:18.026771 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:18.027123 (+ 352us) server_negotiation.cc:299] Negotiation successful
0618 22:18:18.027227 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260618 22:18:18.027971 7487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:18.013139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:39017 (local address 127.0.98.124:36495)
0618 22:18:18.013270 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:18.013276 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:18.013370 (+ 94us) server_negotiation.cc:407] Connection header received
0618 22:18:18.013827 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:18.013831 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:18.013885 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:18.013958 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:18.015516 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:18.016277 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:18.018240 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:18.018597 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:18.018698 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:18.022635 (+ 3937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:18.022640 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:18.022646 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:18.022675 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:18.024529 (+ 1854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:18.025183 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:18.025187 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:18.025189 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:18.025275 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:18.025664 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:18.025669 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:18.025671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:18.025923 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:18.026022 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:18.027718 (+ 1696us) server_negotiation.cc:299] Negotiation successful
0618 22:18:18.027832 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260618 22:18:18.028589 7760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:18.012985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:36495 (local address 127.0.98.67:39017)
0618 22:18:18.013270 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:18.013284 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:18.013583 (+ 299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:18.014116 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:18.014122 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:18.014596 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:18.015344 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:18.015364 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:18.016454 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:18.016457 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:18.018119 (+ 1662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:18.018129 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:18.018883 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:18.018887 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:18.019084 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:18.019797 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:18.019815 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:18.022396 (+ 2581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:18:18.024661 (+ 2265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:18.024670 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:18.024688 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:18.025065 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:18.025381 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:18.025386 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:18.025388 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:18.025557 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:18.026039 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:18:18.026045 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:18:18.026342 (+ 297us) client_negotiation.cc:769] Sending connection context
0618 22:18:18.026802 (+ 460us) client_negotiation.cc:240] Negotiation successful
0618 22:18:18.027151 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":258,"spinlock_wait_cycles":5888,"thread_start_us":91,"threads_started":1}
I20260618 22:18:18.028864 6553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:18.011665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:34739 (local address 127.0.98.126:42267)
0618 22:18:18.011817 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:18.011822 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:18.012672 (+ 850us) server_negotiation.cc:407] Connection header received
0618 22:18:18.013785 (+ 1113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:18.013791 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:18.013847 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:18.013948 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:18.015513 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:18.016329 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:18.020922 (+ 4593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:18.021263 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:18.021392 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:18.024577 (+ 3185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:18.024581 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:18.024585 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:18.024612 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:18.026724 (+ 2112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:18.027223 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:18.027227 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:18.027229 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:18.027310 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:18.027679 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:18.027685 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:18.027688 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:18.027899 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:18.027991 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:18.028584 (+ 593us) server_negotiation.cc:299] Negotiation successful
0618 22:18:18.028708 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260618 22:18:18.028914 7762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:18.012010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:34675 (local address 127.0.98.67:59315)
0618 22:18:18.013229 (+ 1219us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:18.013243 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:18.013570 (+ 327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:18.014254 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:18.014259 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:18.014634 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:18.015351 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:18.015364 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:18.016469 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:18.016473 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:18.018740 (+ 2267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:18.018748 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:18.019704 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:18.019709 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:18.019827 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:18.020290 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:18.020309 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:18.022834 (+ 2525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:18:18.025641 (+ 2807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:18.025646 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:18.025648 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:18.025881 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:18.026245 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:18.026248 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:18.026250 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:18.026342 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:18.026826 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:18:18.026829 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:18:18.026913 (+ 84us) client_negotiation.cc:769] Sending connection context
0618 22:18:18.027574 (+ 661us) client_negotiation.cc:240] Negotiation successful
0618 22:18:18.027705 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1132,"mutex_wait_us":1445,"spinlock_wait_cycles":43008,"thread_start_us":133,"threads_started":1}
I20260618 22:18:18.029227 7761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:18.012010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:42267 (local address 127.0.98.67:34739)
0618 22:18:18.012540 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:18.012577 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:18.013570 (+ 993us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:18.014115 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:18.014121 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:18.014607 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:18.015349 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:18.015364 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:18.016469 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:18.016473 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:18.020778 (+ 4305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:18.020787 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:18.021410 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:18.021414 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:18.021518 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:18.022380 (+ 862us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:18.022394 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:18.024410 (+ 2016us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:18:18.026851 (+ 2441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:18.026854 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:18.026856 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:18.027108 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:18.027453 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:18.027456 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:18.027458 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:18.027562 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:18.028036 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:18:18.028076 (+ 40us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:18:18.028181 (+ 105us) client_negotiation.cc:769] Sending connection context
0618 22:18:18.028280 (+ 99us) client_negotiation.cc:240] Negotiation successful
0618 22:18:18.028408 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"mutex_wait_us":2966,"spinlock_wait_cycles":12544,"thread_start_us":122,"threads_started":1}
I20260618 22:18:18.029546 7770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:18.017457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36462 (local address 127.0.98.124:36495)
0618 22:18:18.017677 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:18.017687 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:18.018722 (+ 1035us) server_negotiation.cc:407] Connection header received
0618 22:18:18.018767 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:18.018771 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:18.018819 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:18.018900 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:18.019715 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:18.020499 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:18.022801 (+ 2302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:18.023186 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:18.023301 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:18.025224 (+ 1923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:18.025230 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:18.025232 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:18.025261 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:18.027345 (+ 2084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:18.027857 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:18.027861 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:18.027862 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:18.027942 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:18.028275 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:18.028279 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:18.028280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:18.028483 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:18.028596 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:18.029279 (+ 683us) server_negotiation.cc:299] Negotiation successful
0618 22:18:18.029399 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":81,"threads_started":1}
I20260618 22:18:18.030197 7758 heartbeater.cc:347] Connected to a master server at 127.0.98.125:34675
I20260618 22:18:18.030310 7757 heartbeater.cc:347] Connected to a master server at 127.0.98.126:42267
I20260618 22:18:18.030444 7758 heartbeater.cc:464] Registering TS with master...
I20260618 22:18:18.030469 7756 heartbeater.cc:347] Connected to a master server at 127.0.98.124:36495
I20260618 22:18:18.030576 7756 heartbeater.cc:464] Registering TS with master...
I20260618 22:18:18.031034 7756 heartbeater.cc:511] Master 127.0.98.124:36495 requested a full tablet report, sending...
I20260618 22:18:18.031036 7758 heartbeater.cc:511] Master 127.0.98.125:34675 requested a full tablet report, sending...
I20260618 22:18:18.030442 7757 heartbeater.cc:464] Registering TS with master...
I20260618 22:18:18.031764 7757 heartbeater.cc:511] Master 127.0.98.126:42267 requested a full tablet report, sending...
I20260618 22:18:18.032183 7115 ts_manager.cc:194] Registered new tserver with Master: 5256bda72de544e985aaad69508c7748 (127.0.98.67:46203)
I20260618 22:18:18.032189 7295 ts_manager.cc:194] Registered new tserver with Master: 5256bda72de544e985aaad69508c7748 (127.0.98.67:46203)
I20260618 22:18:18.032290 6514 ts_manager.cc:194] Re-registered known tserver with Master: 5256bda72de544e985aaad69508c7748 (127.0.98.67:46203)
I20260618 22:18:18.032958 6514 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.67@KRBTEST.COM'} at 127.0.98.67:34739
I20260618 22:18:18.037084 7768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:18.017056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59752 (local address 127.0.98.126:42267)
0618 22:18:18.018357 (+ 1301us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:18.018370 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:18.018382 (+ 12us) server_negotiation.cc:407] Connection header received
0618 22:18:18.018420 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:18.018422 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:18.018466 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:18.018552 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:18.019551 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:18.020383 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:18.023581 (+ 3198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:18.023959 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:18.024057 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:18.033698 (+ 9641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:18.033701 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:18.033704 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:18.033727 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:18.035343 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:18.035772 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:18.035778 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:18.035780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:18.035867 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:18.036147 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:18.036150 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:18.036152 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:18.036391 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:18.036533 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:18.036776 (+ 243us) server_negotiation.cc:299] Negotiation successful
0618 22:18:18.036895 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1211,"thread_start_us":61,"threads_started":1}
I20260618 22:18:18.039256 7772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:18.017960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54080 (local address 127.0.98.125:34675)
0618 22:18:18.021021 (+ 3061us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:18.021025 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:18.025832 (+ 4807us) server_negotiation.cc:407] Connection header received
0618 22:18:18.025874 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:18.025877 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:18.025949 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:18.030446 (+ 4497us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:18.031509 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:18.033049 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:18.034137 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:18.034514 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:18.034628 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:18.035843 (+ 1215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:18.035846 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:18.035849 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:18.035871 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:18.037456 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:18.037868 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:18.037872 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:18.037873 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:18.037997 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:18.038314 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:18.038319 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:18.038322 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:18.038545 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:18.038663 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:18.038904 (+ 241us) server_negotiation.cc:299] Negotiation successful
0618 22:18:18.039052 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2994,"thread_start_us":70,"threads_started":1}
I20260618 22:18:18.039878 6514 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59752:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 22:18:18.042688 6514 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:18:18.044910 7073 sys_catalog.cc:455] T 00000000000000000000000000000000 P cafe178b94854b17acaae5da06c1920d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cafe178b94854b17acaae5da06c1920d" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "cafe178b94854b17acaae5da06c1920d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42267 } } peers { permanent_uuid: "512c63ff46bd41d6a51ca7e026cc8d47" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 34675 } attrs { promote: false } } peers { permanent_uuid: "9d8bc0e14290456db37208ee7d35b1cc" member_type: NON_VOTER last_known_addr { host: "127.0.98.124" port: 36495 } attrs { promote: true } } }
I20260618 22:18:18.045042 7073 sys_catalog.cc:458] T 00000000000000000000000000000000 P cafe178b94854b17acaae5da06c1920d [sys.catalog]: This master's current role is: LEADER
I20260618 22:18:18.046622 7073 raft_consensus.cc:1050] T 00000000000000000000000000000000 P cafe178b94854b17acaae5da06c1920d: attempting to promote NON_VOTER 9d8bc0e14290456db37208ee7d35b1cc to VOTER
I20260618 22:18:18.046967 7073 consensus_queue.cc:237] T 00000000000000000000000000000000 P cafe178b94854b17acaae5da06c1920d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cafe178b94854b17acaae5da06c1920d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42267 } } peers { permanent_uuid: "512c63ff46bd41d6a51ca7e026cc8d47" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 34675 } attrs { promote: false } } peers { permanent_uuid: "9d8bc0e14290456db37208ee7d35b1cc" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 36495 } attrs { promote: false } }
I20260618 22:18:18.047988 7125 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 512c63ff46bd41d6a51ca7e026cc8d47 [term 1 FOLLOWER]: Refusing update from remote peer cafe178b94854b17acaae5da06c1920d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260618 22:18:18.048350 7774 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cafe178b94854b17acaae5da06c1920d [LEADER]: Connected to new peer: Peer: permanent_uuid: "512c63ff46bd41d6a51ca7e026cc8d47" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 34675 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260618 22:18:18.049803 7776 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cafe178b94854b17acaae5da06c1920d [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 9d8bc0e14290456db37208ee7d35b1cc (127.0.98.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cafe178b94854b17acaae5da06c1920d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42267 } } peers { permanent_uuid: "512c63ff46bd41d6a51ca7e026cc8d47" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 34675 } attrs { promote: false } } peers { permanent_uuid: "9d8bc0e14290456db37208ee7d35b1cc" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 36495 } attrs { promote: false } } }
I20260618 22:18:18.050240 7314 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc [term 1 LEARNER]: Refusing update from remote peer cafe178b94854b17acaae5da06c1920d: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260618 22:18:18.050225 7125 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 512c63ff46bd41d6a51ca7e026cc8d47 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 9d8bc0e14290456db37208ee7d35b1cc (127.0.98.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cafe178b94854b17acaae5da06c1920d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42267 } } peers { permanent_uuid: "512c63ff46bd41d6a51ca7e026cc8d47" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 34675 } attrs { promote: false } } peers { permanent_uuid: "9d8bc0e14290456db37208ee7d35b1cc" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 36495 } attrs { promote: false } } }
I20260618 22:18:18.050609 7073 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cafe178b94854b17acaae5da06c1920d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9d8bc0e14290456db37208ee7d35b1cc" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 36495 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20260618 22:18:18.051589 7314 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 9d8bc0e14290456db37208ee7d35b1cc (127.0.98.124) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cafe178b94854b17acaae5da06c1920d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42267 } } peers { permanent_uuid: "512c63ff46bd41d6a51ca7e026cc8d47" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 34675 } attrs { promote: false } } peers { permanent_uuid: "9d8bc0e14290456db37208ee7d35b1cc" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 36495 } attrs { promote: false } } }
I20260618 22:18:18.051748 7073 sys_catalog.cc:455] T 00000000000000000000000000000000 P cafe178b94854b17acaae5da06c1920d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "cafe178b94854b17acaae5da06c1920d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cafe178b94854b17acaae5da06c1920d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42267 } } peers { permanent_uuid: "512c63ff46bd41d6a51ca7e026cc8d47" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 34675 } attrs { promote: false } } peers { permanent_uuid: "9d8bc0e14290456db37208ee7d35b1cc" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 36495 } attrs { promote: false } } }
I20260618 22:18:18.051846 7073 sys_catalog.cc:458] T 00000000000000000000000000000000 P cafe178b94854b17acaae5da06c1920d [sys.catalog]: This master's current role is: LEADER
I20260618 22:18:18.052042 7073 sys_catalog.cc:455] T 00000000000000000000000000000000 P cafe178b94854b17acaae5da06c1920d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cafe178b94854b17acaae5da06c1920d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cafe178b94854b17acaae5da06c1920d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42267 } } peers { permanent_uuid: "512c63ff46bd41d6a51ca7e026cc8d47" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 34675 } attrs { promote: false } } peers { permanent_uuid: "9d8bc0e14290456db37208ee7d35b1cc" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 36495 } attrs { promote: false } } }
I20260618 22:18:18.052140 7073 sys_catalog.cc:458] T 00000000000000000000000000000000 P cafe178b94854b17acaae5da06c1920d [sys.catalog]: This master's current role is: LEADER
I20260618 22:18:18.053046 7346 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cafe178b94854b17acaae5da06c1920d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cafe178b94854b17acaae5da06c1920d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42267 } } peers { permanent_uuid: "512c63ff46bd41d6a51ca7e026cc8d47" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 34675 } attrs { promote: false } } peers { permanent_uuid: "9d8bc0e14290456db37208ee7d35b1cc" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 36495 } attrs { promote: false } } }
I20260618 22:18:18.053157 7346 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:18:18.056072 7784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 512c63ff46bd41d6a51ca7e026cc8d47 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cafe178b94854b17acaae5da06c1920d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cafe178b94854b17acaae5da06c1920d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42267 } } peers { permanent_uuid: "512c63ff46bd41d6a51ca7e026cc8d47" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 34675 } attrs { promote: false } } peers { permanent_uuid: "9d8bc0e14290456db37208ee7d35b1cc" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 36495 } attrs { promote: false } } }
I20260618 22:18:18.056197 7784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 512c63ff46bd41d6a51ca7e026cc8d47 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:18:18.056794 7776 sys_catalog.cc:455] T 00000000000000000000000000000000 P cafe178b94854b17acaae5da06c1920d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cafe178b94854b17acaae5da06c1920d" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cafe178b94854b17acaae5da06c1920d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42267 } } peers { permanent_uuid: "512c63ff46bd41d6a51ca7e026cc8d47" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 34675 } attrs { promote: false } } peers { permanent_uuid: "9d8bc0e14290456db37208ee7d35b1cc" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 36495 } attrs { promote: false } } }
I20260618 22:18:18.056910 7776 sys_catalog.cc:458] T 00000000000000000000000000000000 P cafe178b94854b17acaae5da06c1920d [sys.catalog]: This master's current role is: LEADER
I20260618 22:18:18.066221 7071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:18.061089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:34925 (local address 127.0.0.1:37666)
0618 22:18:18.061656 (+ 567us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:18.061673 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:18.061746 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:18.062630 (+ 884us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:18.062633 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:18.062650 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:18:18.062756 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:18.062767 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:18.063894 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:18.063901 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:18.064766 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:18.064775 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:18.065841 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:18.065845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:18.065962 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:18.065992 (+ 30us) client_negotiation.cc:769] Sending connection context
0618 22:18:18.066042 (+ 50us) client_negotiation.cc:240] Negotiation successful
0618 22:18:18.066103 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492}
I20260618 22:18:18.066686 7788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:18.061269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:41835 (local address 127.0.0.1:51852)
0618 22:18:18.061790 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:18.061805 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:18.061889 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:18.062866 (+ 977us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:18.062869 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:18.062884 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:18:18.062969 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:18.062975 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:18.064514 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:18.064518 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:18.065305 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:18.065313 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:18.066359 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:18.066364 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:18.066456 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:18.066483 (+ 27us) client_negotiation.cc:769] Sending connection context
0618 22:18:18.066530 (+ 47us) client_negotiation.cc:240] Negotiation successful
0618 22:18:18.066586 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"mutex_wait_us":16,"thread_start_us":117,"threads_started":1}
I20260618 22:18:18.067114 7789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:18.061934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37666 (local address 127.0.98.65:34925)
0618 22:18:18.062174 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:18.062186 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:18.062203 (+ 17us) server_negotiation.cc:407] Connection header received
0618 22:18:18.062263 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:18.062268 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:18.062475 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:18.062660 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:18:18.062902 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:18.063771 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:18.064900 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:18.065697 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:18.066654 (+ 957us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:18.066811 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:18.066865 (+ 54us) server_negotiation.cc:299] Negotiation successful
0618 22:18:18.066958 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":75,"threads_started":1}
I20260618 22:18:18.067305 7786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:18.061093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:46203 (local address 127.0.0.1:41282)
0618 22:18:18.062189 (+ 1096us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:18.062202 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:18.062324 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:18.063015 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:18.063021 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:18.063037 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:18:18.063122 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:18.063132 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:18.064492 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:18.064495 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:18.065671 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:18.065681 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:18.066909 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:18.066913 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:18.066997 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:18.067019 (+ 22us) client_negotiation.cc:769] Sending connection context
0618 22:18:18.067088 (+ 69us) client_negotiation.cc:240] Negotiation successful
0618 22:18:18.067163 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1039,"mutex_wait_us":447,"thread_start_us":78,"threads_started":1}
I20260618 22:18:18.067395 7787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:18.061471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51852 (local address 127.0.98.66:41835)
0618 22:18:18.062107 (+ 636us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:18.062120 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:18.062142 (+ 22us) server_negotiation.cc:407] Connection header received
0618 22:18:18.062221 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:18.062228 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:18.062456 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:18.063189 (+ 733us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:18:18.063334 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:18.064385 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:18.065425 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:18.066210 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:18.066970 (+ 760us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:18.067078 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:18.067116 (+ 38us) server_negotiation.cc:299] Negotiation successful
0618 22:18:18.067238 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":106,"threads_started":1}
I20260618 22:18:18.067759 7785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:18.061218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41282 (local address 127.0.98.67:46203)
0618 22:18:18.061885 (+ 667us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:18.061931 (+ 46us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:18.062304 (+ 373us) server_negotiation.cc:407] Connection header received
0618 22:18:18.062487 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:18.062493 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:18.062704 (+ 211us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:18.062867 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:18:18.063244 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:18.064372 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:18.065826 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:18.066644 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:18.066734 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:18.066851 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:18.067544 (+ 693us) server_negotiation.cc:299] Negotiation successful
0618 22:18:18.067634 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":127,"threads_started":1}
I20260618 22:18:18.070072 7554 tablet_service.cc:1467] Processing CreateTablet for tablet f647c809367543c9b8f0415eda7f42e4 (DEFAULT_TABLE table=test-table [id=17bf81fe39bd47d5922109f4d92214b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:18:18.070072 7690 tablet_service.cc:1467] Processing CreateTablet for tablet f647c809367543c9b8f0415eda7f42e4 (DEFAULT_TABLE table=test-table [id=17bf81fe39bd47d5922109f4d92214b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:18:18.070026 7417 tablet_service.cc:1467] Processing CreateTablet for tablet f647c809367543c9b8f0415eda7f42e4 (DEFAULT_TABLE table=test-table [id=17bf81fe39bd47d5922109f4d92214b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:18:18.071058 7417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f647c809367543c9b8f0415eda7f42e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:18:18.071056 7690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f647c809367543c9b8f0415eda7f42e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:18:18.071067 7554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f647c809367543c9b8f0415eda7f42e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:18:18.077488 7790 tablet_bootstrap.cc:492] T f647c809367543c9b8f0415eda7f42e4 P 4c35cb6b49cd461b83b7ea8f6e4dcb54: Bootstrap starting.
I20260618 22:18:18.079273 7792 tablet_bootstrap.cc:492] T f647c809367543c9b8f0415eda7f42e4 P d7dc899699d84381a27dbdf8dae05cc2: Bootstrap starting.
I20260618 22:18:18.079274 7791 tablet_bootstrap.cc:492] T f647c809367543c9b8f0415eda7f42e4 P 5256bda72de544e985aaad69508c7748: Bootstrap starting.
I20260618 22:18:18.079986 7790 tablet_bootstrap.cc:654] T f647c809367543c9b8f0415eda7f42e4 P 4c35cb6b49cd461b83b7ea8f6e4dcb54: Neither blocks nor log segments found. Creating new log.
I20260618 22:18:18.080729 7790 log.cc:824] T f647c809367543c9b8f0415eda7f42e4 P 4c35cb6b49cd461b83b7ea8f6e4dcb54: Log is configured to *not* fsync() on all Append() calls
I20260618 22:18:18.081396 7792 tablet_bootstrap.cc:654] T f647c809367543c9b8f0415eda7f42e4 P d7dc899699d84381a27dbdf8dae05cc2: Neither blocks nor log segments found. Creating new log.
I20260618 22:18:18.081396 7791 tablet_bootstrap.cc:654] T f647c809367543c9b8f0415eda7f42e4 P 5256bda72de544e985aaad69508c7748: Neither blocks nor log segments found. Creating new log.
I20260618 22:18:18.082479 7790 tablet_bootstrap.cc:492] T f647c809367543c9b8f0415eda7f42e4 P 4c35cb6b49cd461b83b7ea8f6e4dcb54: No bootstrap required, opened a new log
I20260618 22:18:18.082602 7791 log.cc:824] T f647c809367543c9b8f0415eda7f42e4 P 5256bda72de544e985aaad69508c7748: Log is configured to *not* fsync() on all Append() calls
I20260618 22:18:18.082602 7792 log.cc:824] T f647c809367543c9b8f0415eda7f42e4 P d7dc899699d84381a27dbdf8dae05cc2: Log is configured to *not* fsync() on all Append() calls
I20260618 22:18:18.082696 7790 ts_tablet_manager.cc:1397] T f647c809367543c9b8f0415eda7f42e4 P 4c35cb6b49cd461b83b7ea8f6e4dcb54: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260618 22:18:18.084431 7792 tablet_bootstrap.cc:492] T f647c809367543c9b8f0415eda7f42e4 P d7dc899699d84381a27dbdf8dae05cc2: No bootstrap required, opened a new log
I20260618 22:18:18.084591 7792 ts_tablet_manager.cc:1397] T f647c809367543c9b8f0415eda7f42e4 P d7dc899699d84381a27dbdf8dae05cc2: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260618 22:18:18.084810 7791 tablet_bootstrap.cc:492] T f647c809367543c9b8f0415eda7f42e4 P 5256bda72de544e985aaad69508c7748: No bootstrap required, opened a new log
I20260618 22:18:18.084959 7791 ts_tablet_manager.cc:1397] T f647c809367543c9b8f0415eda7f42e4 P 5256bda72de544e985aaad69508c7748: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260618 22:18:18.086962 7790 raft_consensus.cc:348] T f647c809367543c9b8f0415eda7f42e4 P 4c35cb6b49cd461b83b7ea8f6e4dcb54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c35cb6b49cd461b83b7ea8f6e4dcb54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34925 } } peers { permanent_uuid: "d7dc899699d84381a27dbdf8dae05cc2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 41835 } } peers { permanent_uuid: "5256bda72de544e985aaad69508c7748" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46203 } }
I20260618 22:18:18.087224 7790 raft_consensus.cc:374] T f647c809367543c9b8f0415eda7f42e4 P 4c35cb6b49cd461b83b7ea8f6e4dcb54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:18:18.087280 7790 raft_consensus.cc:729] T f647c809367543c9b8f0415eda7f42e4 P 4c35cb6b49cd461b83b7ea8f6e4dcb54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c35cb6b49cd461b83b7ea8f6e4dcb54, State: Initialized, Role: FOLLOWER
I20260618 22:18:18.087706 7790 consensus_queue.cc:260] T f647c809367543c9b8f0415eda7f42e4 P 4c35cb6b49cd461b83b7ea8f6e4dcb54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c35cb6b49cd461b83b7ea8f6e4dcb54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34925 } } peers { permanent_uuid: "d7dc899699d84381a27dbdf8dae05cc2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 41835 } } peers { permanent_uuid: "5256bda72de544e985aaad69508c7748" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46203 } }
I20260618 22:18:18.088666 7790 ts_tablet_manager.cc:1428] T f647c809367543c9b8f0415eda7f42e4 P 4c35cb6b49cd461b83b7ea8f6e4dcb54: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260618 22:18:18.088827 7484 heartbeater.cc:503] Master 127.0.98.126:42267 was elected leader, sending a full tablet report...
I20260618 22:18:18.089362 7792 raft_consensus.cc:348] T f647c809367543c9b8f0415eda7f42e4 P d7dc899699d84381a27dbdf8dae05cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c35cb6b49cd461b83b7ea8f6e4dcb54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34925 } } peers { permanent_uuid: "d7dc899699d84381a27dbdf8dae05cc2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 41835 } } peers { permanent_uuid: "5256bda72de544e985aaad69508c7748" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46203 } }
I20260618 22:18:18.089362 7791 raft_consensus.cc:348] T f647c809367543c9b8f0415eda7f42e4 P 5256bda72de544e985aaad69508c7748 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c35cb6b49cd461b83b7ea8f6e4dcb54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34925 } } peers { permanent_uuid: "d7dc899699d84381a27dbdf8dae05cc2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 41835 } } peers { permanent_uuid: "5256bda72de544e985aaad69508c7748" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46203 } }
I20260618 22:18:18.089596 7791 raft_consensus.cc:374] T f647c809367543c9b8f0415eda7f42e4 P 5256bda72de544e985aaad69508c7748 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:18:18.089596 7792 raft_consensus.cc:374] T f647c809367543c9b8f0415eda7f42e4 P d7dc899699d84381a27dbdf8dae05cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:18:18.089643 7792 raft_consensus.cc:729] T f647c809367543c9b8f0415eda7f42e4 P d7dc899699d84381a27dbdf8dae05cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7dc899699d84381a27dbdf8dae05cc2, State: Initialized, Role: FOLLOWER
I20260618 22:18:18.089643 7791 raft_consensus.cc:729] T f647c809367543c9b8f0415eda7f42e4 P 5256bda72de544e985aaad69508c7748 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5256bda72de544e985aaad69508c7748, State: Initialized, Role: FOLLOWER
I20260618 22:18:18.090081 7791 consensus_queue.cc:260] T f647c809367543c9b8f0415eda7f42e4 P 5256bda72de544e985aaad69508c7748 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c35cb6b49cd461b83b7ea8f6e4dcb54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34925 } } peers { permanent_uuid: "d7dc899699d84381a27dbdf8dae05cc2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 41835 } } peers { permanent_uuid: "5256bda72de544e985aaad69508c7748" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46203 } }
I20260618 22:18:18.090081 7792 consensus_queue.cc:260] T f647c809367543c9b8f0415eda7f42e4 P d7dc899699d84381a27dbdf8dae05cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c35cb6b49cd461b83b7ea8f6e4dcb54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34925 } } peers { permanent_uuid: "d7dc899699d84381a27dbdf8dae05cc2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 41835 } } peers { permanent_uuid: "5256bda72de544e985aaad69508c7748" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46203 } }
I20260618 22:18:18.090955 7791 ts_tablet_manager.cc:1428] T f647c809367543c9b8f0415eda7f42e4 P 5256bda72de544e985aaad69508c7748: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260618 22:18:18.090955 7792 ts_tablet_manager.cc:1428] T f647c809367543c9b8f0415eda7f42e4 P d7dc899699d84381a27dbdf8dae05cc2: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260618 22:18:18.092113 7757 heartbeater.cc:503] Master 127.0.98.126:42267 was elected leader, sending a full tablet report...
I20260618 22:18:18.094583 7621 heartbeater.cc:503] Master 127.0.98.126:42267 was elected leader, sending a full tablet report...
W20260618 22:18:18.126048 7486 tablet.cc:2369] T f647c809367543c9b8f0415eda7f42e4 P 4c35cb6b49cd461b83b7ea8f6e4dcb54: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260618 22:18:18.261792 7759 tablet.cc:2369] T f647c809367543c9b8f0415eda7f42e4 P 5256bda72de544e985aaad69508c7748: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:18:18.307648 7796 raft_consensus.cc:482] T f647c809367543c9b8f0415eda7f42e4 P 4c35cb6b49cd461b83b7ea8f6e4dcb54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:18:18.307834 7796 raft_consensus.cc:504] T f647c809367543c9b8f0415eda7f42e4 P 4c35cb6b49cd461b83b7ea8f6e4dcb54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c35cb6b49cd461b83b7ea8f6e4dcb54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34925 } } peers { permanent_uuid: "d7dc899699d84381a27dbdf8dae05cc2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 41835 } } peers { permanent_uuid: "5256bda72de544e985aaad69508c7748" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46203 } }
I20260618 22:18:18.308880 7796 leader_election.cc:288] T f647c809367543c9b8f0415eda7f42e4 P 4c35cb6b49cd461b83b7ea8f6e4dcb54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7dc899699d84381a27dbdf8dae05cc2 (127.0.98.66:41835), 5256bda72de544e985aaad69508c7748 (127.0.98.67:46203)
I20260618 22:18:18.312711 7785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:18.309214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:38049 (local address 127.0.98.67:46203)
0618 22:18:18.309360 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:18.309364 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:18.309631 (+ 267us) server_negotiation.cc:407] Connection header received
0618 22:18:18.309750 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:18.309753 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:18.309805 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:18.309884 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:18:18.310206 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:18.310681 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:18.311765 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:18.312175 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:18.312294 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:18.312357 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:18.312511 (+ 154us) server_negotiation.cc:299] Negotiation successful
0618 22:18:18.312585 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260618 22:18:18.312739 7787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:18.309329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:43193 (local address 127.0.98.66:41835)
0618 22:18:18.309442 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:18.309446 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:18.309576 (+ 130us) server_negotiation.cc:407] Connection header received
0618 22:18:18.309703 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:18.309708 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:18.309768 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:18.309826 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:18:18.310141 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:18.310650 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:18.311791 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:18.312179 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:18.312305 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:18.312362 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:18.312571 (+ 209us) server_negotiation.cc:299] Negotiation successful
0618 22:18:18.312636 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260618 22:18:18.312728 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:18.309120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:46203 (local address 127.0.98.65:38049)
0618 22:18:18.309493 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:18.309506 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:18.309643 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:18.309936 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:18.309940 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:18.309959 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:18:18.310065 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:18.310069 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:18.310814 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:18.310817 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:18.311640 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:18.311648 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:18.312297 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:18.312300 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:18.312386 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:18.312407 (+ 21us) client_negotiation.cc:769] Sending connection context
0618 22:18:18.312476 (+ 69us) client_negotiation.cc:240] Negotiation successful
0618 22:18:18.312556 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"mutex_wait_us":175,"thread_start_us":61,"threads_started":1}
I20260618 22:18:18.312752 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:18.309120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:41835 (local address 127.0.98.65:43193)
0618 22:18:18.309476 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:18.309489 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:18.309608 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:18.309861 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:18.309866 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:18.309886 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:18:18.310038 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:18.310044 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:18.310768 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:18.310775 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:18.311639 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:18.311648 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:18.312326 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:18.312329 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:18.312432 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:18.312449 (+ 17us) client_negotiation.cc:769] Sending connection context
0618 22:18:18.312546 (+ 97us) client_negotiation.cc:240] Negotiation successful
0618 22:18:18.312607 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":286,"mutex_wait_us":35,"thread_start_us":84,"threads_started":1}
I20260618 22:18:18.313421 7710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f647c809367543c9b8f0415eda7f42e4" candidate_uuid: "4c35cb6b49cd461b83b7ea8f6e4dcb54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5256bda72de544e985aaad69508c7748" is_pre_election: true
I20260618 22:18:18.313426 7574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f647c809367543c9b8f0415eda7f42e4" candidate_uuid: "4c35cb6b49cd461b83b7ea8f6e4dcb54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7dc899699d84381a27dbdf8dae05cc2" is_pre_election: true
I20260618 22:18:18.313699 7574 raft_consensus.cc:2454] T f647c809367543c9b8f0415eda7f42e4 P d7dc899699d84381a27dbdf8dae05cc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c35cb6b49cd461b83b7ea8f6e4dcb54 in term 0.
I20260618 22:18:18.313706 7710 raft_consensus.cc:2454] T f647c809367543c9b8f0415eda7f42e4 P 5256bda72de544e985aaad69508c7748 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c35cb6b49cd461b83b7ea8f6e4dcb54 in term 0.
I20260618 22:18:18.314571 7373 leader_election.cc:302] T f647c809367543c9b8f0415eda7f42e4 P 4c35cb6b49cd461b83b7ea8f6e4dcb54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c35cb6b49cd461b83b7ea8f6e4dcb54, d7dc899699d84381a27dbdf8dae05cc2; no voters:
I20260618 22:18:18.314833 7796 raft_consensus.cc:2781] T f647c809367543c9b8f0415eda7f42e4 P 4c35cb6b49cd461b83b7ea8f6e4dcb54 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:18:18.314903 7796 raft_consensus.cc:482] T f647c809367543c9b8f0415eda7f42e4 P 4c35cb6b49cd461b83b7ea8f6e4dcb54 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:18:18.314955 7796 raft_consensus.cc:3037] T f647c809367543c9b8f0415eda7f42e4 P 4c35cb6b49cd461b83b7ea8f6e4dcb54 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:18:18.316120 7796 raft_consensus.cc:504] T f647c809367543c9b8f0415eda7f42e4 P 4c35cb6b49cd461b83b7ea8f6e4dcb54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c35cb6b49cd461b83b7ea8f6e4dcb54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34925 } } peers { permanent_uuid: "d7dc899699d84381a27dbdf8dae05cc2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 41835 } } peers { permanent_uuid: "5256bda72de544e985aaad69508c7748" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46203 } }
I20260618 22:18:18.316464 7796 leader_election.cc:288] T f647c809367543c9b8f0415eda7f42e4 P 4c35cb6b49cd461b83b7ea8f6e4dcb54 [CANDIDATE]: Term 1 election: Requested vote from peers d7dc899699d84381a27dbdf8dae05cc2 (127.0.98.66:41835), 5256bda72de544e985aaad69508c7748 (127.0.98.67:46203)
I20260618 22:18:18.316813 7710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f647c809367543c9b8f0415eda7f42e4" candidate_uuid: "4c35cb6b49cd461b83b7ea8f6e4dcb54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5256bda72de544e985aaad69508c7748"
I20260618 22:18:18.316937 7710 raft_consensus.cc:3037] T f647c809367543c9b8f0415eda7f42e4 P 5256bda72de544e985aaad69508c7748 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:18:18.318095 7710 raft_consensus.cc:2454] T f647c809367543c9b8f0415eda7f42e4 P 5256bda72de544e985aaad69508c7748 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c35cb6b49cd461b83b7ea8f6e4dcb54 in term 1.
I20260618 22:18:18.318071 7574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f647c809367543c9b8f0415eda7f42e4" candidate_uuid: "4c35cb6b49cd461b83b7ea8f6e4dcb54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7dc899699d84381a27dbdf8dae05cc2"
I20260618 22:18:18.318198 7574 raft_consensus.cc:3037] T f647c809367543c9b8f0415eda7f42e4 P d7dc899699d84381a27dbdf8dae05cc2 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:18:18.318368 7372 leader_election.cc:302] T f647c809367543c9b8f0415eda7f42e4 P 4c35cb6b49cd461b83b7ea8f6e4dcb54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c35cb6b49cd461b83b7ea8f6e4dcb54, 5256bda72de544e985aaad69508c7748; no voters:
I20260618 22:18:18.318542 7796 raft_consensus.cc:2781] T f647c809367543c9b8f0415eda7f42e4 P 4c35cb6b49cd461b83b7ea8f6e4dcb54 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:18:18.318786 7796 raft_consensus.cc:686] T f647c809367543c9b8f0415eda7f42e4 P 4c35cb6b49cd461b83b7ea8f6e4dcb54 [term 1 LEADER]: Becoming Leader. State: Replica: 4c35cb6b49cd461b83b7ea8f6e4dcb54, State: Running, Role: LEADER
I20260618 22:18:18.319453 7574 raft_consensus.cc:2454] T f647c809367543c9b8f0415eda7f42e4 P d7dc899699d84381a27dbdf8dae05cc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c35cb6b49cd461b83b7ea8f6e4dcb54 in term 1.
I20260618 22:18:18.319432 7796 consensus_queue.cc:237] T f647c809367543c9b8f0415eda7f42e4 P 4c35cb6b49cd461b83b7ea8f6e4dcb54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c35cb6b49cd461b83b7ea8f6e4dcb54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34925 } } peers { permanent_uuid: "d7dc899699d84381a27dbdf8dae05cc2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 41835 } } peers { permanent_uuid: "5256bda72de544e985aaad69508c7748" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46203 } }
W20260618 22:18:18.320809 7623 tablet.cc:2369] T f647c809367543c9b8f0415eda7f42e4 P d7dc899699d84381a27dbdf8dae05cc2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:18:18.322685 6512 catalog_manager.cc:5495] T f647c809367543c9b8f0415eda7f42e4 P 4c35cb6b49cd461b83b7ea8f6e4dcb54 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c35cb6b49cd461b83b7ea8f6e4dcb54 (127.0.98.65). New cstate: current_term: 1 leader_uuid: "4c35cb6b49cd461b83b7ea8f6e4dcb54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c35cb6b49cd461b83b7ea8f6e4dcb54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7dc899699d84381a27dbdf8dae05cc2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 41835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5256bda72de544e985aaad69508c7748" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46203 } health_report { overall_health: UNKNOWN } } }
I20260618 22:18:18.358901 7789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:18.355438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37676 (local address 127.0.98.65:34925)
0618 22:18:18.355554 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:18.355558 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:18.355581 (+ 23us) server_negotiation.cc:407] Connection header received
0618 22:18:18.355650 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:18.355652 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:18.355702 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:18.355800 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:18:18.355987 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:18.356487 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:18.357329 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:18.357534 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:18.357612 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:18.357872 (+ 260us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:18:18.358457 (+ 585us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:18:18.358552 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:18.358711 (+ 159us) server_negotiation.cc:299] Negotiation successful
0618 22:18:18.358777 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260618 22:18:18.365309 7574 raft_consensus.cc:1261] T f647c809367543c9b8f0415eda7f42e4 P d7dc899699d84381a27dbdf8dae05cc2 [term 1 FOLLOWER]: Refusing update from remote peer 4c35cb6b49cd461b83b7ea8f6e4dcb54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:18:18.365309 7710 raft_consensus.cc:1261] T f647c809367543c9b8f0415eda7f42e4 P 5256bda72de544e985aaad69508c7748 [term 1 FOLLOWER]: Refusing update from remote peer 4c35cb6b49cd461b83b7ea8f6e4dcb54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:18:18.366139 7801 consensus_queue.cc:1035] T f647c809367543c9b8f0415eda7f42e4 P 4c35cb6b49cd461b83b7ea8f6e4dcb54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7dc899699d84381a27dbdf8dae05cc2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 41835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:18:18.366312 7796 consensus_queue.cc:1035] T f647c809367543c9b8f0415eda7f42e4 P 4c35cb6b49cd461b83b7ea8f6e4dcb54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5256bda72de544e985aaad69508c7748" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:18:18.375344 7806 mvcc.cc:205] Tried to move back new op lower bound from 7298339218897383424 to 7298339218719227904. Current Snapshot: MvccSnapshot[applied={T|T < 7298339218897383424}]
I20260618 22:18:18.377689 7805 mvcc.cc:205] Tried to move back new op lower bound from 7298339218897383424 to 7298339218719227904. Current Snapshot: MvccSnapshot[applied={T|T < 7298339218897383424}]
I20260618 22:18:18.378311 7804 mvcc.cc:205] Tried to move back new op lower bound from 7298339218897383424 to 7298339218719227904. Current Snapshot: MvccSnapshot[applied={T|T < 7298339218897383424 or (T in {7298339218897383424})}]
I20260618 22:18:18.383702 6512 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59752:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:18:18.383903 6512 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59752:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:18:18.387784 6512 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cafe178b94854b17acaae5da06c1920d: Sending DeleteTablet for 3 replicas of tablet f647c809367543c9b8f0415eda7f42e4
I20260618 22:18:18.388760 7417 tablet_service.cc:1514] Processing DeleteTablet for tablet f647c809367543c9b8f0415eda7f42e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:18:18 UTC) from {username='oryx', principal='oryx/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:37666
I20260618 22:18:18.388784 7554 tablet_service.cc:1514] Processing DeleteTablet for tablet f647c809367543c9b8f0415eda7f42e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:18:18 UTC) from {username='oryx', principal='oryx/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:51852
I20260618 22:18:18.388885 7690 tablet_service.cc:1514] Processing DeleteTablet for tablet f647c809367543c9b8f0415eda7f42e4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:18:18 UTC) from {username='oryx', principal='oryx/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:41282
I20260618 22:18:18.389424 7814 tablet_replica.cc:330] T f647c809367543c9b8f0415eda7f42e4 P 4c35cb6b49cd461b83b7ea8f6e4dcb54: stopping tablet replica
I20260618 22:18:18.389750 7814 raft_consensus.cc:2229] T f647c809367543c9b8f0415eda7f42e4 P 4c35cb6b49cd461b83b7ea8f6e4dcb54 [term 1 LEADER]: Raft consensus shutting down.
I20260618 22:18:18.389814 7815 tablet_replica.cc:330] T f647c809367543c9b8f0415eda7f42e4 P 5256bda72de544e985aaad69508c7748: stopping tablet replica
I20260618 22:18:18.390208 7814 raft_consensus.cc:2258] T f647c809367543c9b8f0415eda7f42e4 P 4c35cb6b49cd461b83b7ea8f6e4dcb54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:18:18.390355 7813 tablet_replica.cc:330] T f647c809367543c9b8f0415eda7f42e4 P d7dc899699d84381a27dbdf8dae05cc2: stopping tablet replica
I20260618 22:18:18.390653 7813 raft_consensus.cc:2229] T f647c809367543c9b8f0415eda7f42e4 P d7dc899699d84381a27dbdf8dae05cc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:18:18.390970 7813 raft_consensus.cc:2258] T f647c809367543c9b8f0415eda7f42e4 P d7dc899699d84381a27dbdf8dae05cc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:18:18.391391 7815 raft_consensus.cc:2229] T f647c809367543c9b8f0415eda7f42e4 P 5256bda72de544e985aaad69508c7748 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:18:18.391543 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 6482
I20260618 22:18:18.392204 7815 raft_consensus.cc:2258] T f647c809367543c9b8f0415eda7f42e4 P 5256bda72de544e985aaad69508c7748 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:18:18.393191 7814 ts_tablet_manager.cc:1905] T f647c809367543c9b8f0415eda7f42e4 P 4c35cb6b49cd461b83b7ea8f6e4dcb54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:18:18.393191 7813 ts_tablet_manager.cc:1905] T f647c809367543c9b8f0415eda7f42e4 P d7dc899699d84381a27dbdf8dae05cc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:18:18.393631 7815 ts_tablet_manager.cc:1905] T f647c809367543c9b8f0415eda7f42e4 P 5256bda72de544e985aaad69508c7748: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:18:18.396154 7813 ts_tablet_manager.cc:1918] T f647c809367543c9b8f0415eda7f42e4 P d7dc899699d84381a27dbdf8dae05cc2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:18:18.396265 7813 log.cc:1196] T f647c809367543c9b8f0415eda7f42e4 P d7dc899699d84381a27dbdf8dae05cc2: Deleting WAL directory at /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-1/wal/wals/f647c809367543c9b8f0415eda7f42e4
I20260618 22:18:18.396387 7814 ts_tablet_manager.cc:1918] T f647c809367543c9b8f0415eda7f42e4 P 4c35cb6b49cd461b83b7ea8f6e4dcb54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:18:18.396541 7814 log.cc:1196] T f647c809367543c9b8f0415eda7f42e4 P 4c35cb6b49cd461b83b7ea8f6e4dcb54: Deleting WAL directory at /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-0/wal/wals/f647c809367543c9b8f0415eda7f42e4
I20260618 22:18:18.396642 7813 ts_tablet_manager.cc:1939] T f647c809367543c9b8f0415eda7f42e4 P d7dc899699d84381a27dbdf8dae05cc2: Deleting consensus metadata
I20260618 22:18:18.396988 7814 ts_tablet_manager.cc:1939] T f647c809367543c9b8f0415eda7f42e4 P 4c35cb6b49cd461b83b7ea8f6e4dcb54: Deleting consensus metadata
I20260618 22:18:18.397121 7815 ts_tablet_manager.cc:1918] T f647c809367543c9b8f0415eda7f42e4 P 5256bda72de544e985aaad69508c7748: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:18:18.397220 7815 log.cc:1196] T f647c809367543c9b8f0415eda7f42e4 P 5256bda72de544e985aaad69508c7748: Deleting WAL directory at /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-2/wal/wals/f647c809367543c9b8f0415eda7f42e4
I20260618 22:18:18.397543 7815 ts_tablet_manager.cc:1939] T f647c809367543c9b8f0415eda7f42e4 P 5256bda72de544e985aaad69508c7748: Deleting consensus metadata
W20260618 22:18:18.398840 7510 connection.cc:535] server connection from 127.0.0.1:51852 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260618 22:18:18.398878 7646 connection.cc:535] server connection from 127.0.0.1:41282 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260618 22:18:18.398945 7370 connection.cc:535] server connection from 127.0.0.1:37666 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260618 22:18:18.411249 7772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:18.390480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54086 (local address 127.0.98.125:34675)
0618 22:18:18.390597 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:18.390602 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:18.398473 (+ 7871us) server_negotiation.cc:407] Connection header received
0618 22:18:18.398622 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:18.398625 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:18.398692 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:18.398787 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:18.400039 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:18.400889 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:18.402272 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:18.402645 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:18.402759 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:18.404938 (+ 2179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:18.404942 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:18.404945 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:18.404971 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:18.407224 (+ 2253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:18.408223 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:18.408227 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:18.408230 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:18.408320 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:18.408979 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:18.408986 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:18.408992 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:18.409772 (+ 780us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:18.409989 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:18.410720 (+ 731us) server_negotiation.cc:299] Negotiation successful
0618 22:18:18.411037 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260618 22:18:18.412235 7487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:18.402665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36474 (local address 127.0.98.124:36495)
0618 22:18:18.402786 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:18.402791 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:18.402805 (+ 14us) server_negotiation.cc:407] Connection header received
0618 22:18:18.402849 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:18.402853 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:18.402910 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:18.402995 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:18.404129 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:18.404897 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:18.406102 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:18.406437 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:18.406505 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:18.408259 (+ 1754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:18.408263 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:18.408266 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:18.408290 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:18.410021 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:18.410469 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:18.410473 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:18.410475 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:18.410549 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:18.410833 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:18.410836 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:18.410838 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:18.411022 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:18.411114 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:18.411982 (+ 868us) server_negotiation.cc:299] Negotiation successful
0618 22:18:18.412091 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260618 22:18:18.413628 7237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:18.402902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54088 (local address 127.0.98.125:34675)
0618 22:18:18.403022 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:18.403026 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:18.403179 (+ 153us) server_negotiation.cc:407] Connection header received
0618 22:18:18.403352 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:18.403357 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:18.403416 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:18.403472 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:18.404548 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:18.405367 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:18.406800 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:18.407211 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:18.407300 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:18.409032 (+ 1732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:18.409038 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:18.409044 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:18.409066 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:18.410882 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:18.412056 (+ 1174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:18.412061 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:18.412065 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:18.412164 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:18.412725 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:18.412729 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:18.412731 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:18.412925 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:18.413019 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:18.413389 (+ 370us) server_negotiation.cc:299] Negotiation successful
0618 22:18:18.413492 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260618 22:18:18.420158 7770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:18.391158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36472 (local address 127.0.98.124:36495)
0618 22:18:18.391274 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:18.391278 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:18.410988 (+ 19710us) server_negotiation.cc:407] Connection header received
0618 22:18:18.411160 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:18.411164 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:18.411210 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:18.411264 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:18.412381 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:18.413182 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:18.414529 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:18.414794 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:18.414915 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:18.416539 (+ 1624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:18.416543 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:18.416546 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:18.416571 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:18.418294 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:18.418739 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:18.418745 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:18.418747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:18.418829 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:18.419121 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:18.419125 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:18.419126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:18.419322 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:18.419453 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:18.419890 (+ 437us) server_negotiation.cc:299] Negotiation successful
0618 22:18:18.420013 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260618 22:18:19.377017 7823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:19.376553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:42267 (local address 127.0.98.66:57741)
0618 22:18:19.376823 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:19.376927 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:42267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":80,"threads_started":1}
W20260618 22:18:19.377321 7621 heartbeater.cc:644] Failed to heartbeat to 127.0.98.126:42267 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.126:42267: connect: Connection refused (error 111)
I20260618 22:18:19.378746 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:19.378142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:42267 (local address 127.0.98.67:38237)
0618 22:18:19.378559 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:19.378653 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:42267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":108,"threads_started":1}
W20260618 22:18:19.379245 7757 heartbeater.cc:644] Failed to heartbeat to 127.0.98.126:42267 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.126:42267: connect: Connection refused (error 111)
I20260618 22:18:19.379339 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:19.378733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:42267 (local address 127.0.98.65:45359)
0618 22:18:19.379153 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:19.379230 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:42267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":59,"threads_started":1}
W20260618 22:18:19.379626 7484 heartbeater.cc:644] Failed to heartbeat to 127.0.98.126:42267 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.126:42267: connect: Connection refused (error 111)
I20260618 22:18:19.888309 7826 raft_consensus.cc:482] T 00000000000000000000000000000000 P 512c63ff46bd41d6a51ca7e026cc8d47 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cafe178b94854b17acaae5da06c1920d)
I20260618 22:18:19.888482 7826 raft_consensus.cc:504] T 00000000000000000000000000000000 P 512c63ff46bd41d6a51ca7e026cc8d47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cafe178b94854b17acaae5da06c1920d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42267 } } peers { permanent_uuid: "512c63ff46bd41d6a51ca7e026cc8d47" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 34675 } attrs { promote: false } } peers { permanent_uuid: "9d8bc0e14290456db37208ee7d35b1cc" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 36495 } attrs { promote: false } }
I20260618 22:18:19.888991 7827 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc [term 1 FOLLOWER]: Starting pre-election (detected failure of leader cafe178b94854b17acaae5da06c1920d)
I20260618 22:18:19.889166 7827 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cafe178b94854b17acaae5da06c1920d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42267 } } peers { permanent_uuid: "512c63ff46bd41d6a51ca7e026cc8d47" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 34675 } attrs { promote: false } } peers { permanent_uuid: "9d8bc0e14290456db37208ee7d35b1cc" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 36495 } attrs { promote: false } }
I20260618 22:18:19.889523 7826 leader_election.cc:288] T 00000000000000000000000000000000 P 512c63ff46bd41d6a51ca7e026cc8d47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cafe178b94854b17acaae5da06c1920d (127.0.98.126:42267), 9d8bc0e14290456db37208ee7d35b1cc (127.0.98.124:36495)
I20260618 22:18:19.890158 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:19.889694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:42267 (local address 127.0.0.1:60216)
0618 22:18:19.889949 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:19.890043 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:42267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":79,"threads_started":1}
W20260618 22:18:19.890377 7101 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.126:42267: connect: Connection refused (error 111)
I20260618 22:18:19.890849 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:19.890227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:42267 (local address 127.0.0.1:60230)
0618 22:18:19.890691 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:19.890761 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:42267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":51,"threads_started":1}
W20260618 22:18:19.891075 7290 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.126:42267: connect: Connection refused (error 111)
I20260618 22:18:19.891215 7828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:19.890997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:42267 (local address 127.0.0.1:60240)
0618 22:18:19.891086 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:19.891143 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:42267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260618 22:18:19.891326 7101 leader_election.cc:334] T 00000000000000000000000000000000 P 512c63ff46bd41d6a51ca7e026cc8d47 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer cafe178b94854b17acaae5da06c1920d (127.0.98.126:42267): Network error: Client connection negotiation failed: client connection to 127.0.98.126:42267: connect: Connection refused (error 111)
I20260618 22:18:19.891367 7827 leader_election.cc:288] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cafe178b94854b17acaae5da06c1920d (127.0.98.126:42267), 512c63ff46bd41d6a51ca7e026cc8d47 (127.0.98.125:34675)
I20260618 22:18:19.891955 7833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:19.891736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:42267 (local address 127.0.0.1:60246)
0618 22:18:19.891822 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:19.891882 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:42267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260618 22:18:19.892131 7290 leader_election.cc:334] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer cafe178b94854b17acaae5da06c1920d (127.0.98.126:42267): Network error: Client connection negotiation failed: client connection to 127.0.98.126:42267: connect: Connection refused (error 111)
I20260618 22:18:19.894747 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:19.890048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:36495 (local address 127.0.0.1:36480)
0618 22:18:19.890717 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:19.890731 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:19.890866 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:19.891739 (+ 873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:19.891742 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:19.891754 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:18:19.891833 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:19.891838 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:19.892664 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:19.892667 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:19.893486 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:19.893495 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:19.894324 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:19.894328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:19.894418 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:19.894457 (+ 39us) client_negotiation.cc:769] Sending connection context
0618 22:18:19.894526 (+ 69us) client_negotiation.cc:240] Negotiation successful
0618 22:18:19.894600 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":3,"thread_start_us":62,"threads_started":1}
I20260618 22:18:19.894764 7831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:19.890405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36480 (local address 127.0.98.124:36495)
0618 22:18:19.890638 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:19.890643 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:19.890788 (+ 145us) server_negotiation.cc:407] Connection header received
0618 22:18:19.891550 (+ 762us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:19.891555 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:19.891607 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:19.891679 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:18:19.891948 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:19.892566 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:19.893608 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:19.894193 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:19.894329 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:19.894389 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:19.894545 (+ 156us) server_negotiation.cc:299] Negotiation successful
0618 22:18:19.894640 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":173,"server-negotiator.queue_time_us":159,"thread_start_us":53,"threads_started":1}
I20260618 22:18:19.894881 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:19.890768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:34675 (local address 127.0.0.1:54104)
0618 22:18:19.890991 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:19.891004 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:19.891101 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:19.891651 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:19.891656 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:19.891682 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:18:19.891807 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:19.891814 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:19.892825 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:19.892827 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:19.893680 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:19.893688 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:19.894464 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:19.894468 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:19.894577 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:19.894596 (+ 19us) client_negotiation.cc:769] Sending connection context
0618 22:18:19.894679 (+ 83us) client_negotiation.cc:240] Negotiation successful
0618 22:18:19.894734 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":3,"thread_start_us":77,"threads_started":1}
I20260618 22:18:19.894908 7834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:19.891099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54104 (local address 127.0.98.125:34675)
0618 22:18:19.891374 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:19.891384 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:19.891400 (+ 16us) server_negotiation.cc:407] Connection header received
0618 22:18:19.891446 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:19.891450 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:19.891493 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:19.891610 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:18:19.892120 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:19.892723 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:19.893815 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:19.894362 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:19.894457 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:19.894517 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:19.894717 (+ 200us) server_negotiation.cc:299] Negotiation successful
0618 22:18:19.894786 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":88,"threads_started":1}
I20260618 22:18:19.895525 7125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9d8bc0e14290456db37208ee7d35b1cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "512c63ff46bd41d6a51ca7e026cc8d47" is_pre_election: true
I20260618 22:18:19.895712 7125 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 512c63ff46bd41d6a51ca7e026cc8d47 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d8bc0e14290456db37208ee7d35b1cc in term 1.
I20260618 22:18:19.895705 7314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "512c63ff46bd41d6a51ca7e026cc8d47" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9d8bc0e14290456db37208ee7d35b1cc" is_pre_election: true
I20260618 22:18:19.895897 7314 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 512c63ff46bd41d6a51ca7e026cc8d47 in term 1.
I20260618 22:18:19.896020 7288 leader_election.cc:302] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 512c63ff46bd41d6a51ca7e026cc8d47, 9d8bc0e14290456db37208ee7d35b1cc; no voters: cafe178b94854b17acaae5da06c1920d
I20260618 22:18:19.896209 7827 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260618 22:18:19.896200 7101 leader_election.cc:302] T 00000000000000000000000000000000 P 512c63ff46bd41d6a51ca7e026cc8d47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 512c63ff46bd41d6a51ca7e026cc8d47, 9d8bc0e14290456db37208ee7d35b1cc; no voters: cafe178b94854b17acaae5da06c1920d
I20260618 22:18:19.896265 7827 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc [term 1 FOLLOWER]: Starting leader election (detected failure of leader cafe178b94854b17acaae5da06c1920d)
I20260618 22:18:19.896312 7827 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc [term 1 FOLLOWER]: Advancing to term 2
I20260618 22:18:19.896396 7826 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 512c63ff46bd41d6a51ca7e026cc8d47 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260618 22:18:19.896449 7826 raft_consensus.cc:482] T 00000000000000000000000000000000 P 512c63ff46bd41d6a51ca7e026cc8d47 [term 1 FOLLOWER]: Starting leader election (detected failure of leader cafe178b94854b17acaae5da06c1920d)
I20260618 22:18:19.896477 7826 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 512c63ff46bd41d6a51ca7e026cc8d47 [term 1 FOLLOWER]: Advancing to term 2
I20260618 22:18:19.897289 7827 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cafe178b94854b17acaae5da06c1920d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42267 } } peers { permanent_uuid: "512c63ff46bd41d6a51ca7e026cc8d47" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 34675 } attrs { promote: false } } peers { permanent_uuid: "9d8bc0e14290456db37208ee7d35b1cc" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 36495 } attrs { promote: false } }
I20260618 22:18:19.897344 7826 raft_consensus.cc:504] T 00000000000000000000000000000000 P 512c63ff46bd41d6a51ca7e026cc8d47 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cafe178b94854b17acaae5da06c1920d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42267 } } peers { permanent_uuid: "512c63ff46bd41d6a51ca7e026cc8d47" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 34675 } attrs { promote: false } } peers { permanent_uuid: "9d8bc0e14290456db37208ee7d35b1cc" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 36495 } attrs { promote: false } }
I20260618 22:18:19.897677 7826 leader_election.cc:288] T 00000000000000000000000000000000 P 512c63ff46bd41d6a51ca7e026cc8d47 [CANDIDATE]: Term 2 election: Requested vote from peers cafe178b94854b17acaae5da06c1920d (127.0.98.126:42267), 9d8bc0e14290456db37208ee7d35b1cc (127.0.98.124:36495)
I20260618 22:18:19.897938 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:19.897699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:42267 (local address 127.0.0.1:60254)
0618 22:18:19.897800 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:19.897852 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:42267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260618 22:18:19.898077 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:19.897720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:42267 (local address 127.0.0.1:60260)
0618 22:18:19.897970 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:19.898019 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:42267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
I20260618 22:18:19.898124 7125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9d8bc0e14290456db37208ee7d35b1cc" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "512c63ff46bd41d6a51ca7e026cc8d47"
I20260618 22:18:19.898154 7314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "512c63ff46bd41d6a51ca7e026cc8d47" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "9d8bc0e14290456db37208ee7d35b1cc"
I20260618 22:18:19.898269 7125 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 512c63ff46bd41d6a51ca7e026cc8d47 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 9d8bc0e14290456db37208ee7d35b1cc in current term 2: Already voted for candidate 512c63ff46bd41d6a51ca7e026cc8d47 in this term.
I20260618 22:18:19.898317 7314 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 512c63ff46bd41d6a51ca7e026cc8d47 in current term 2: Already voted for candidate 9d8bc0e14290456db37208ee7d35b1cc in this term.
I20260618 22:18:19.898834 7830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:19.898617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:42267 (local address 127.0.0.1:60262)
0618 22:18:19.898725 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:19.898777 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:42267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260618 22:18:19.898962 7290 leader_election.cc:334] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer cafe178b94854b17acaae5da06c1920d (127.0.98.126:42267): Network error: Client connection negotiation failed: client connection to 127.0.98.126:42267: connect: Connection refused (error 111)
I20260618 22:18:19.898991 7829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:19.898795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:42267 (local address 127.0.0.1:60274)
0618 22:18:19.898892 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:19.898938 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:42267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260618 22:18:19.899010 7827 leader_election.cc:288] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc [CANDIDATE]: Term 2 election: Requested vote from peers cafe178b94854b17acaae5da06c1920d (127.0.98.126:42267), 512c63ff46bd41d6a51ca7e026cc8d47 (127.0.98.125:34675)
I20260618 22:18:19.899022 7290 leader_election.cc:302] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9d8bc0e14290456db37208ee7d35b1cc; no voters: 512c63ff46bd41d6a51ca7e026cc8d47, cafe178b94854b17acaae5da06c1920d
I20260618 22:18:19.899173 7827 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260618 22:18:19.899284 7101 leader_election.cc:334] T 00000000000000000000000000000000 P 512c63ff46bd41d6a51ca7e026cc8d47 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer cafe178b94854b17acaae5da06c1920d (127.0.98.126:42267): Network error: Client connection negotiation failed: client connection to 127.0.98.126:42267: connect: Connection refused (error 111)
I20260618 22:18:19.899355 7101 leader_election.cc:302] T 00000000000000000000000000000000 P 512c63ff46bd41d6a51ca7e026cc8d47 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 512c63ff46bd41d6a51ca7e026cc8d47; no voters: 9d8bc0e14290456db37208ee7d35b1cc, cafe178b94854b17acaae5da06c1920d
I20260618 22:18:19.899470 7826 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 512c63ff46bd41d6a51ca7e026cc8d47 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260618 22:18:20.378403 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:20.377935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:42267 (local address 127.0.98.66:57199)
0618 22:18:20.378211 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:20.378283 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:42267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":78,"threads_started":1}
I20260618 22:18:20.380395 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:20.379807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:42267 (local address 127.0.98.67:44003)
0618 22:18:20.380202 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:20.380270 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:42267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":86,"threads_started":1}
I20260618 22:18:20.380822 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:20.380111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:42267 (local address 127.0.98.65:50225)
0618 22:18:20.380695 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:20.380752 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:42267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":78,"threads_started":1}
I20260618 22:18:21.379758 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:21.379237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:42267 (local address 127.0.98.66:40611)
0618 22:18:21.379560 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:21.379628 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:42267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":78,"threads_started":1}
I20260618 22:18:21.381502 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:21.381072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:42267 (local address 127.0.98.67:34101)
0618 22:18:21.381347 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:21.381411 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:42267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":65,"threads_started":1}
I20260618 22:18:21.382059 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:21.381490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:42267 (local address 127.0.98.65:35553)
0618 22:18:21.381885 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:21.381972 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:42267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":68,"threads_started":1}
I20260618 22:18:21.694988 7842 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:18:21.695124 7842 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cafe178b94854b17acaae5da06c1920d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42267 } } peers { permanent_uuid: "512c63ff46bd41d6a51ca7e026cc8d47" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 34675 } attrs { promote: false } } peers { permanent_uuid: "9d8bc0e14290456db37208ee7d35b1cc" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 36495 } attrs { promote: false } }
I20260618 22:18:21.695559 7842 leader_election.cc:288] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers cafe178b94854b17acaae5da06c1920d (127.0.98.126:42267), 512c63ff46bd41d6a51ca7e026cc8d47 (127.0.98.125:34675)
I20260618 22:18:21.696007 7125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9d8bc0e14290456db37208ee7d35b1cc" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "512c63ff46bd41d6a51ca7e026cc8d47" is_pre_election: true
I20260618 22:18:21.696136 7125 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 512c63ff46bd41d6a51ca7e026cc8d47 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d8bc0e14290456db37208ee7d35b1cc in term 2.
I20260618 22:18:21.696260 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:21.695785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:42267 (local address 127.0.0.1:60510)
0618 22:18:21.696026 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:21.696114 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:42267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":72,"threads_started":1}
I20260618 22:18:21.696532 7288 leader_election.cc:302] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 512c63ff46bd41d6a51ca7e026cc8d47, 9d8bc0e14290456db37208ee7d35b1cc; no voters:
I20260618 22:18:21.696913 7842 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260618 22:18:21.696983 7842 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:18:21.697039 7842 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc [term 2 FOLLOWER]: Advancing to term 3
I20260618 22:18:21.697063 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:21.696807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:42267 (local address 127.0.0.1:60512)
0618 22:18:21.696917 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:21.696971 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:42267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260618 22:18:21.697131 7290 leader_election.cc:334] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer cafe178b94854b17acaae5da06c1920d (127.0.98.126:42267): Network error: Client connection negotiation failed: client connection to 127.0.98.126:42267: connect: Connection refused (error 111)
I20260618 22:18:21.698110 7842 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cafe178b94854b17acaae5da06c1920d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42267 } } peers { permanent_uuid: "512c63ff46bd41d6a51ca7e026cc8d47" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 34675 } attrs { promote: false } } peers { permanent_uuid: "9d8bc0e14290456db37208ee7d35b1cc" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 36495 } attrs { promote: false } }
I20260618 22:18:21.698427 7842 leader_election.cc:288] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc [CANDIDATE]: Term 3 election: Requested vote from peers cafe178b94854b17acaae5da06c1920d (127.0.98.126:42267), 512c63ff46bd41d6a51ca7e026cc8d47 (127.0.98.125:34675)
I20260618 22:18:21.698699 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:21.698493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:42267 (local address 127.0.0.1:60528)
0618 22:18:21.698614 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:21.698652 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:42267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260618 22:18:21.698807 7125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9d8bc0e14290456db37208ee7d35b1cc" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "512c63ff46bd41d6a51ca7e026cc8d47"
I20260618 22:18:21.698927 7125 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 512c63ff46bd41d6a51ca7e026cc8d47 [term 2 FOLLOWER]: Advancing to term 3
I20260618 22:18:21.699383 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:21.699216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:42267 (local address 127.0.0.1:60536)
0618 22:18:21.699301 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:21.699341 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:42267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260618 22:18:21.699534 7290 leader_election.cc:334] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer cafe178b94854b17acaae5da06c1920d (127.0.98.126:42267): Network error: Client connection negotiation failed: client connection to 127.0.98.126:42267: connect: Connection refused (error 111)
I20260618 22:18:21.700039 7125 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 512c63ff46bd41d6a51ca7e026cc8d47 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d8bc0e14290456db37208ee7d35b1cc in term 3.
I20260618 22:18:21.700296 7288 leader_election.cc:302] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 512c63ff46bd41d6a51ca7e026cc8d47, 9d8bc0e14290456db37208ee7d35b1cc; no voters: cafe178b94854b17acaae5da06c1920d
I20260618 22:18:21.700532 7842 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc [term 3 FOLLOWER]: Leader election won for term 3
I20260618 22:18:21.700742 7842 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc [term 3 LEADER]: Becoming Leader. State: Replica: 9d8bc0e14290456db37208ee7d35b1cc, State: Running, Role: LEADER
I20260618 22:18:21.700994 7842 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cafe178b94854b17acaae5da06c1920d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42267 } } peers { permanent_uuid: "512c63ff46bd41d6a51ca7e026cc8d47" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 34675 } attrs { promote: false } } peers { permanent_uuid: "9d8bc0e14290456db37208ee7d35b1cc" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 36495 } attrs { promote: false } }
I20260618 22:18:21.702061 7845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d8bc0e14290456db37208ee7d35b1cc. Latest consensus state: current_term: 3 leader_uuid: "9d8bc0e14290456db37208ee7d35b1cc" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cafe178b94854b17acaae5da06c1920d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42267 } } peers { permanent_uuid: "512c63ff46bd41d6a51ca7e026cc8d47" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 34675 } attrs { promote: false } } peers { permanent_uuid: "9d8bc0e14290456db37208ee7d35b1cc" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 36495 } attrs { promote: false } } }
I20260618 22:18:21.702198 7845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc [sys.catalog]: This master's current role is: LEADER
I20260618 22:18:21.702500 7847 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:18:21.704447 7847 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:18:21.704840 7847 catalog_manager.cc:1259] Loaded cluster ID: fc2c6ba654e446178226c06313f3379d
I20260618 22:18:21.704882 7847 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:18:21.705442 7847 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:18:21.705829 7847 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc: Loaded TSK: 0
I20260618 22:18:21.706496 7847 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:18:21.788975 7298 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36474:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 22:18:21.792049 7298 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:18:21.795130 7125 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 512c63ff46bd41d6a51ca7e026cc8d47 [term 3 FOLLOWER]: Refusing update from remote peer 9d8bc0e14290456db37208ee7d35b1cc: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260618 22:18:21.795207 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:21.794943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:42267 (local address 127.0.0.1:60550)
0618 22:18:21.795063 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:21.795120 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:42267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260618 22:18:21.795723 7845 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc [LEADER]: Connected to new peer: Peer: permanent_uuid: "512c63ff46bd41d6a51ca7e026cc8d47" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 34675 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260618 22:18:21.795861 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:21.795619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:42267 (local address 127.0.0.1:60558)
0618 22:18:21.795751 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:21.795799 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:42267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260618 22:18:21.795953 7290 consensus_peers.cc:487] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc -> Peer cafe178b94854b17acaae5da06c1920d (127.0.98.126:42267): Couldn't send request to peer cafe178b94854b17acaae5da06c1920d. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.126:42267: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260618 22:18:21.797050 7849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 512c63ff46bd41d6a51ca7e026cc8d47 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d8bc0e14290456db37208ee7d35b1cc. Latest consensus state: current_term: 3 leader_uuid: "9d8bc0e14290456db37208ee7d35b1cc" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cafe178b94854b17acaae5da06c1920d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42267 } } peers { permanent_uuid: "512c63ff46bd41d6a51ca7e026cc8d47" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 34675 } attrs { promote: false } } peers { permanent_uuid: "9d8bc0e14290456db37208ee7d35b1cc" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 36495 } attrs { promote: false } } }
I20260618 22:18:21.797188 7849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 512c63ff46bd41d6a51ca7e026cc8d47 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:18:21.798285 7845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "9d8bc0e14290456db37208ee7d35b1cc" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cafe178b94854b17acaae5da06c1920d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42267 } } peers { permanent_uuid: "512c63ff46bd41d6a51ca7e026cc8d47" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 34675 } attrs { promote: false } } peers { permanent_uuid: "9d8bc0e14290456db37208ee7d35b1cc" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 36495 } attrs { promote: false } } }
I20260618 22:18:21.798419 7845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc [sys.catalog]: This master's current role is: LEADER
I20260618 22:18:21.798812 7850 mvcc.cc:205] Tried to move back new op lower bound from 7298339232947564544 to 7298339232569954304. Current Snapshot: MvccSnapshot[applied={T|T < 7298339232947564544}]
I20260618 22:18:21.799382 7849 sys_catalog.cc:455] T 00000000000000000000000000000000 P 512c63ff46bd41d6a51ca7e026cc8d47 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "9d8bc0e14290456db37208ee7d35b1cc" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "cafe178b94854b17acaae5da06c1920d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42267 } } peers { permanent_uuid: "512c63ff46bd41d6a51ca7e026cc8d47" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 34675 } attrs { promote: false } } peers { permanent_uuid: "9d8bc0e14290456db37208ee7d35b1cc" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 36495 } attrs { promote: false } } }
I20260618 22:18:21.799501 7849 sys_catalog.cc:458] T 00000000000000000000000000000000 P 512c63ff46bd41d6a51ca7e026cc8d47 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:18:21.809032 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:21.804449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37682 (local address 127.0.98.65:34925)
0618 22:18:21.804853 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:21.804858 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:21.804872 (+ 14us) server_negotiation.cc:407] Connection header received
0618 22:18:21.804922 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:21.804925 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:21.804978 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:21.805447 (+ 469us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:18:21.805568 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:21.806376 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:21.807664 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:21.808344 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:21.808479 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:21.808563 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:21.808831 (+ 268us) server_negotiation.cc:299] Negotiation successful
0618 22:18:21.808905 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":62,"threads_started":1}
I20260618 22:18:21.809020 7843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:21.803945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:34925 (local address 127.0.0.1:37682)
0618 22:18:21.804059 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:21.804079 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:21.804168 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:21.805093 (+ 925us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:21.805096 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:21.805110 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:18:21.805227 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:21.805237 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:21.806493 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:21.806496 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:21.807449 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:21.807504 (+ 55us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:21.808523 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:21.808528 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:21.808691 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:21.808711 (+ 20us) client_negotiation.cc:769] Sending connection context
0618 22:18:21.808812 (+ 101us) client_negotiation.cc:240] Negotiation successful
0618 22:18:21.808886 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":65}
I20260618 22:18:21.809435 7857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:21.804443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:46203 (local address 127.0.0.1:41298)
0618 22:18:21.805147 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:21.805161 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:21.805267 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:21.805612 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:21.805615 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:21.805632 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:18:21.805744 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:21.805750 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:21.806604 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:21.806607 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:21.807983 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:21.807993 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:21.808970 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:21.808974 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:21.809137 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:21.809158 (+ 21us) client_negotiation.cc:769] Sending connection context
0618 22:18:21.809223 (+ 65us) client_negotiation.cc:240] Negotiation successful
0618 22:18:21.809288 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":641,"mutex_wait_us":92,"thread_start_us":80,"threads_started":1}
I20260618 22:18:21.809769 7417 tablet_service.cc:1467] Processing CreateTablet for tablet cce659151b544b58affff848b40339ca (DEFAULT_TABLE table=test-table [id=5a3d33af4e404641ac3bec5eaab839ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:18:21.809803 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:21.804782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41298 (local address 127.0.98.67:46203)
0618 22:18:21.805014 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:21.805019 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:21.805357 (+ 338us) server_negotiation.cc:407] Connection header received
0618 22:18:21.805434 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:21.805437 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:21.805490 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:21.805593 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:18:21.805861 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:21.806499 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:21.808128 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:21.808745 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:21.808858 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:21.808909 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:21.809583 (+ 674us) server_negotiation.cc:299] Negotiation successful
0618 22:18:21.809678 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":62,"threads_started":1}
I20260618 22:18:21.809983 7856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:21.804304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51858 (local address 127.0.98.66:41835)
0618 22:18:21.804688 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:21.804694 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:21.804849 (+ 155us) server_negotiation.cc:407] Connection header received
0618 22:18:21.805031 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:21.805036 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:21.805102 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:21.805158 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:18:21.805520 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:21.806370 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:21.808133 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:21.808757 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:21.808841 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:21.808907 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:21.809715 (+ 808us) server_negotiation.cc:299] Negotiation successful
0618 22:18:21.809794 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":94,"threads_started":1}
I20260618 22:18:21.810143 7417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cce659151b544b58affff848b40339ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:18:21.810286 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:21.804188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:41835 (local address 127.0.0.1:51858)
0618 22:18:21.804773 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:21.804787 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:21.804913 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:21.805277 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:21.805281 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:21.805297 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:18:21.805392 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:21.805400 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:21.806514 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:21.806517 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:21.807994 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:21.808000 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:21.808987 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:21.808990 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:21.809575 (+ 585us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:21.809599 (+ 24us) client_negotiation.cc:769] Sending connection context
0618 22:18:21.810108 (+ 509us) client_negotiation.cc:240] Negotiation successful
0618 22:18:21.810182 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"mutex_wait_us":160,"thread_start_us":112,"threads_started":1}
I20260618 22:18:21.810856 7554 tablet_service.cc:1467] Processing CreateTablet for tablet cce659151b544b58affff848b40339ca (DEFAULT_TABLE table=test-table [id=5a3d33af4e404641ac3bec5eaab839ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:18:21.811115 7554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cce659151b544b58affff848b40339ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:18:21.813048 7860 tablet_bootstrap.cc:492] T cce659151b544b58affff848b40339ca P 4c35cb6b49cd461b83b7ea8f6e4dcb54: Bootstrap starting.
I20260618 22:18:21.813760 7690 tablet_service.cc:1467] Processing CreateTablet for tablet cce659151b544b58affff848b40339ca (DEFAULT_TABLE table=test-table [id=5a3d33af4e404641ac3bec5eaab839ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:18:21.813844 7861 tablet_bootstrap.cc:492] T cce659151b544b58affff848b40339ca P d7dc899699d84381a27dbdf8dae05cc2: Bootstrap starting.
I20260618 22:18:21.814029 7860 tablet_bootstrap.cc:654] T cce659151b544b58affff848b40339ca P 4c35cb6b49cd461b83b7ea8f6e4dcb54: Neither blocks nor log segments found. Creating new log.
I20260618 22:18:21.814065 7690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cce659151b544b58affff848b40339ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:18:21.814944 7861 tablet_bootstrap.cc:654] T cce659151b544b58affff848b40339ca P d7dc899699d84381a27dbdf8dae05cc2: Neither blocks nor log segments found. Creating new log.
I20260618 22:18:21.815331 7860 tablet_bootstrap.cc:492] T cce659151b544b58affff848b40339ca P 4c35cb6b49cd461b83b7ea8f6e4dcb54: No bootstrap required, opened a new log
I20260618 22:18:21.815428 7860 ts_tablet_manager.cc:1397] T cce659151b544b58affff848b40339ca P 4c35cb6b49cd461b83b7ea8f6e4dcb54: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260618 22:18:21.815765 7860 raft_consensus.cc:348] T cce659151b544b58affff848b40339ca P 4c35cb6b49cd461b83b7ea8f6e4dcb54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c35cb6b49cd461b83b7ea8f6e4dcb54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34925 } } peers { permanent_uuid: "d7dc899699d84381a27dbdf8dae05cc2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 41835 } } peers { permanent_uuid: "5256bda72de544e985aaad69508c7748" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46203 } }
I20260618 22:18:21.815864 7860 raft_consensus.cc:374] T cce659151b544b58affff848b40339ca P 4c35cb6b49cd461b83b7ea8f6e4dcb54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:18:21.815888 7860 raft_consensus.cc:729] T cce659151b544b58affff848b40339ca P 4c35cb6b49cd461b83b7ea8f6e4dcb54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c35cb6b49cd461b83b7ea8f6e4dcb54, State: Initialized, Role: FOLLOWER
I20260618 22:18:21.816031 7860 consensus_queue.cc:260] T cce659151b544b58affff848b40339ca P 4c35cb6b49cd461b83b7ea8f6e4dcb54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c35cb6b49cd461b83b7ea8f6e4dcb54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34925 } } peers { permanent_uuid: "d7dc899699d84381a27dbdf8dae05cc2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 41835 } } peers { permanent_uuid: "5256bda72de544e985aaad69508c7748" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46203 } }
I20260618 22:18:21.816393 7860 ts_tablet_manager.cc:1428] T cce659151b544b58affff848b40339ca P 4c35cb6b49cd461b83b7ea8f6e4dcb54: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260618 22:18:21.817211 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:21.816801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:42267 (local address 127.0.98.65:41673)
0618 22:18:21.817055 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:21.817110 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:42267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175}
I20260618 22:18:21.817579 7861 tablet_bootstrap.cc:492] T cce659151b544b58affff848b40339ca P d7dc899699d84381a27dbdf8dae05cc2: No bootstrap required, opened a new log
I20260618 22:18:21.817663 7861 ts_tablet_manager.cc:1397] T cce659151b544b58affff848b40339ca P d7dc899699d84381a27dbdf8dae05cc2: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260618 22:18:21.817744 7865 tablet_bootstrap.cc:492] T cce659151b544b58affff848b40339ca P 5256bda72de544e985aaad69508c7748: Bootstrap starting.
I20260618 22:18:21.818131 7861 raft_consensus.cc:348] T cce659151b544b58affff848b40339ca P d7dc899699d84381a27dbdf8dae05cc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c35cb6b49cd461b83b7ea8f6e4dcb54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34925 } } peers { permanent_uuid: "d7dc899699d84381a27dbdf8dae05cc2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 41835 } } peers { permanent_uuid: "5256bda72de544e985aaad69508c7748" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46203 } }
I20260618 22:18:21.818245 7861 raft_consensus.cc:374] T cce659151b544b58affff848b40339ca P d7dc899699d84381a27dbdf8dae05cc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:18:21.818281 7861 raft_consensus.cc:729] T cce659151b544b58affff848b40339ca P d7dc899699d84381a27dbdf8dae05cc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7dc899699d84381a27dbdf8dae05cc2, State: Initialized, Role: FOLLOWER
I20260618 22:18:21.818425 7861 consensus_queue.cc:260] T cce659151b544b58affff848b40339ca P d7dc899699d84381a27dbdf8dae05cc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c35cb6b49cd461b83b7ea8f6e4dcb54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34925 } } peers { permanent_uuid: "d7dc899699d84381a27dbdf8dae05cc2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 41835 } } peers { permanent_uuid: "5256bda72de544e985aaad69508c7748" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46203 } }
I20260618 22:18:21.818526 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:21.818243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:42267 (local address 127.0.98.65:56015)
0618 22:18:21.818393 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:21.818449 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:42267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260618 22:18:21.818751 7861 ts_tablet_manager.cc:1428] T cce659151b544b58affff848b40339ca P d7dc899699d84381a27dbdf8dae05cc2: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260618 22:18:21.819175 7865 tablet_bootstrap.cc:654] T cce659151b544b58affff848b40339ca P 5256bda72de544e985aaad69508c7748: Neither blocks nor log segments found. Creating new log.
I20260618 22:18:21.820134 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:21.819835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:42267 (local address 127.0.98.66:37513)
0618 22:18:21.819979 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:21.820047 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:42267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260618 22:18:21.820741 7865 tablet_bootstrap.cc:492] T cce659151b544b58affff848b40339ca P 5256bda72de544e985aaad69508c7748: No bootstrap required, opened a new log
I20260618 22:18:21.820853 7865 ts_tablet_manager.cc:1397] T cce659151b544b58affff848b40339ca P 5256bda72de544e985aaad69508c7748: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260618 22:18:21.821345 7865 raft_consensus.cc:348] T cce659151b544b58affff848b40339ca P 5256bda72de544e985aaad69508c7748 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c35cb6b49cd461b83b7ea8f6e4dcb54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34925 } } peers { permanent_uuid: "d7dc899699d84381a27dbdf8dae05cc2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 41835 } } peers { permanent_uuid: "5256bda72de544e985aaad69508c7748" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46203 } }
I20260618 22:18:21.821473 7865 raft_consensus.cc:374] T cce659151b544b58affff848b40339ca P 5256bda72de544e985aaad69508c7748 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:18:21.821511 7865 raft_consensus.cc:729] T cce659151b544b58affff848b40339ca P 5256bda72de544e985aaad69508c7748 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5256bda72de544e985aaad69508c7748, State: Initialized, Role: FOLLOWER
I20260618 22:18:21.821542 7620 heartbeater.cc:503] Master 127.0.98.124:36495 was elected leader, sending a full tablet report...
I20260618 22:18:21.821655 7865 consensus_queue.cc:260] T cce659151b544b58affff848b40339ca P 5256bda72de544e985aaad69508c7748 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c35cb6b49cd461b83b7ea8f6e4dcb54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34925 } } peers { permanent_uuid: "d7dc899699d84381a27dbdf8dae05cc2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 41835 } } peers { permanent_uuid: "5256bda72de544e985aaad69508c7748" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46203 } }
I20260618 22:18:21.822065 7865 ts_tablet_manager.cc:1428] T cce659151b544b58affff848b40339ca P 5256bda72de544e985aaad69508c7748: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260618 22:18:21.822798 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:21.822433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:42267 (local address 127.0.98.67:58311)
0618 22:18:21.822659 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:21.822724 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:42267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150}
I20260618 22:18:21.823261 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:21.821346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:42267 (local address 127.0.98.66:55129)
0618 22:18:21.823120 (+ 1774us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:21.823179 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:42267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1700}
I20260618 22:18:21.823477 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:21.823226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:42267 (local address 127.0.98.67:51647)
0618 22:18:21.823372 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:21.823420 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:42267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260618 22:18:21.824476 7483 heartbeater.cc:503] Master 127.0.98.124:36495 was elected leader, sending a full tablet report...
I20260618 22:18:21.824580 7756 heartbeater.cc:503] Master 127.0.98.124:36495 was elected leader, sending a full tablet report...
I20260618 22:18:21.891736 7868 raft_consensus.cc:482] T cce659151b544b58affff848b40339ca P 5256bda72de544e985aaad69508c7748 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:18:21.891865 7868 raft_consensus.cc:504] T cce659151b544b58affff848b40339ca P 5256bda72de544e985aaad69508c7748 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c35cb6b49cd461b83b7ea8f6e4dcb54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34925 } } peers { permanent_uuid: "d7dc899699d84381a27dbdf8dae05cc2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 41835 } } peers { permanent_uuid: "5256bda72de544e985aaad69508c7748" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46203 } }
I20260618 22:18:21.893306 7868 leader_election.cc:288] T cce659151b544b58affff848b40339ca P 5256bda72de544e985aaad69508c7748 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c35cb6b49cd461b83b7ea8f6e4dcb54 (127.0.98.65:34925), d7dc899699d84381a27dbdf8dae05cc2 (127.0.98.66:41835)
I20260618 22:18:21.898164 7858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:21.893646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:42059 (local address 127.0.98.65:34925)
0618 22:18:21.894021 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:21.894025 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:21.894039 (+ 14us) server_negotiation.cc:407] Connection header received
0618 22:18:21.894109 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:21.894112 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:21.894158 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:21.894225 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:18:21.894518 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:21.895317 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:21.896907 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:21.897508 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:21.897609 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:21.897669 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:21.897947 (+ 278us) server_negotiation.cc:299] Negotiation successful
0618 22:18:21.898035 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292}
I20260618 22:18:21.898165 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:21.893560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:34925 (local address 127.0.98.67:42059)
0618 22:18:21.893688 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:21.893701 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:21.893808 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:21.894247 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:21.894252 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:21.894278 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:18:21.894400 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:21.894409 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:21.895414 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:21.895417 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:21.896793 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:21.896800 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:21.897663 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:21.897666 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:21.897797 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:21.897818 (+ 21us) client_negotiation.cc:769] Sending connection context
0618 22:18:21.897882 (+ 64us) client_negotiation.cc:240] Negotiation successful
0618 22:18:21.898004 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":72}
I20260618 22:18:21.898216 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:21.893841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:41835 (local address 127.0.98.67:36945)
0618 22:18:21.894243 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:21.894256 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:21.894340 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:21.894572 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:21.894574 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:21.894591 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:18:21.894693 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:21.894698 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:21.895600 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:21.895603 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:21.896784 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:21.896795 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:21.897742 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:21.897746 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:21.897854 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:21.897906 (+ 52us) client_negotiation.cc:769] Sending connection context
0618 22:18:21.897998 (+ 92us) client_negotiation.cc:240] Negotiation successful
0618 22:18:21.898063 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"mutex_wait_us":45,"thread_start_us":140,"threads_started":1}
I20260618 22:18:21.898248 7856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:21.893959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:36945 (local address 127.0.98.66:41835)
0618 22:18:21.894084 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:21.894088 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:21.894353 (+ 265us) server_negotiation.cc:407] Connection header received
0618 22:18:21.894409 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:21.894411 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:21.894448 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:21.894544 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:18:21.894802 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:21.895508 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:21.896907 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:21.897501 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:21.897609 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:21.897669 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:21.898041 (+ 372us) server_negotiation.cc:299] Negotiation successful
0618 22:18:21.898121 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260618 22:18:21.898797 7574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cce659151b544b58affff848b40339ca" candidate_uuid: "5256bda72de544e985aaad69508c7748" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7dc899699d84381a27dbdf8dae05cc2" is_pre_election: true
I20260618 22:18:21.898830 7437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cce659151b544b58affff848b40339ca" candidate_uuid: "5256bda72de544e985aaad69508c7748" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c35cb6b49cd461b83b7ea8f6e4dcb54" is_pre_election: true
I20260618 22:18:21.898921 7574 raft_consensus.cc:2454] T cce659151b544b58affff848b40339ca P d7dc899699d84381a27dbdf8dae05cc2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5256bda72de544e985aaad69508c7748 in term 0.
I20260618 22:18:21.899070 7437 raft_consensus.cc:2454] T cce659151b544b58affff848b40339ca P 4c35cb6b49cd461b83b7ea8f6e4dcb54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5256bda72de544e985aaad69508c7748 in term 0.
I20260618 22:18:21.899296 7646 leader_election.cc:302] T cce659151b544b58affff848b40339ca P 5256bda72de544e985aaad69508c7748 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5256bda72de544e985aaad69508c7748, d7dc899699d84381a27dbdf8dae05cc2; no voters:
I20260618 22:18:21.899549 7868 raft_consensus.cc:2781] T cce659151b544b58affff848b40339ca P 5256bda72de544e985aaad69508c7748 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:18:21.899612 7868 raft_consensus.cc:482] T cce659151b544b58affff848b40339ca P 5256bda72de544e985aaad69508c7748 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:18:21.899664 7868 raft_consensus.cc:3037] T cce659151b544b58affff848b40339ca P 5256bda72de544e985aaad69508c7748 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:18:21.900681 7868 raft_consensus.cc:504] T cce659151b544b58affff848b40339ca P 5256bda72de544e985aaad69508c7748 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c35cb6b49cd461b83b7ea8f6e4dcb54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34925 } } peers { permanent_uuid: "d7dc899699d84381a27dbdf8dae05cc2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 41835 } } peers { permanent_uuid: "5256bda72de544e985aaad69508c7748" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46203 } }
I20260618 22:18:21.901062 7868 leader_election.cc:288] T cce659151b544b58affff848b40339ca P 5256bda72de544e985aaad69508c7748 [CANDIDATE]: Term 1 election: Requested vote from peers 4c35cb6b49cd461b83b7ea8f6e4dcb54 (127.0.98.65:34925), d7dc899699d84381a27dbdf8dae05cc2 (127.0.98.66:41835)
I20260618 22:18:21.901362 7437 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cce659151b544b58affff848b40339ca" candidate_uuid: "5256bda72de544e985aaad69508c7748" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c35cb6b49cd461b83b7ea8f6e4dcb54"
I20260618 22:18:21.901481 7437 raft_consensus.cc:3037] T cce659151b544b58affff848b40339ca P 4c35cb6b49cd461b83b7ea8f6e4dcb54 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:18:21.901463 7574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cce659151b544b58affff848b40339ca" candidate_uuid: "5256bda72de544e985aaad69508c7748" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7dc899699d84381a27dbdf8dae05cc2"
I20260618 22:18:21.901571 7574 raft_consensus.cc:3037] T cce659151b544b58affff848b40339ca P d7dc899699d84381a27dbdf8dae05cc2 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:18:21.902611 7574 raft_consensus.cc:2454] T cce659151b544b58affff848b40339ca P d7dc899699d84381a27dbdf8dae05cc2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5256bda72de544e985aaad69508c7748 in term 1.
I20260618 22:18:21.902611 7437 raft_consensus.cc:2454] T cce659151b544b58affff848b40339ca P 4c35cb6b49cd461b83b7ea8f6e4dcb54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5256bda72de544e985aaad69508c7748 in term 1.
I20260618 22:18:21.902866 7646 leader_election.cc:302] T cce659151b544b58affff848b40339ca P 5256bda72de544e985aaad69508c7748 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5256bda72de544e985aaad69508c7748, d7dc899699d84381a27dbdf8dae05cc2; no voters:
I20260618 22:18:21.903060 7868 raft_consensus.cc:2781] T cce659151b544b58affff848b40339ca P 5256bda72de544e985aaad69508c7748 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:18:21.903294 7868 raft_consensus.cc:686] T cce659151b544b58affff848b40339ca P 5256bda72de544e985aaad69508c7748 [term 1 LEADER]: Becoming Leader. State: Replica: 5256bda72de544e985aaad69508c7748, State: Running, Role: LEADER
I20260618 22:18:21.903545 7868 consensus_queue.cc:237] T cce659151b544b58affff848b40339ca P 5256bda72de544e985aaad69508c7748 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c35cb6b49cd461b83b7ea8f6e4dcb54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34925 } } peers { permanent_uuid: "d7dc899699d84381a27dbdf8dae05cc2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 41835 } } peers { permanent_uuid: "5256bda72de544e985aaad69508c7748" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46203 } }
I20260618 22:18:21.903892 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:21.903683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:42267 (local address 127.0.98.67:50881)
0618 22:18:21.903788 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:21.903842 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:42267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260618 22:18:21.905745 7297 catalog_manager.cc:5495] T cce659151b544b58affff848b40339ca P 5256bda72de544e985aaad69508c7748 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5256bda72de544e985aaad69508c7748 (127.0.98.67). New cstate: current_term: 1 leader_uuid: "5256bda72de544e985aaad69508c7748" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c35cb6b49cd461b83b7ea8f6e4dcb54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7dc899699d84381a27dbdf8dae05cc2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 41835 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5256bda72de544e985aaad69508c7748" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46203 } health_report { overall_health: HEALTHY } } }
I20260618 22:18:21.941285 7859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:21.937780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41314 (local address 127.0.98.67:46203)
0618 22:18:21.937967 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:21.937974 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:21.937990 (+ 16us) server_negotiation.cc:407] Connection header received
0618 22:18:21.938061 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:21.938063 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:21.938115 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:21.938232 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:18:21.938479 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:21.939000 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:21.939852 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:21.940070 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:21.940182 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:21.940462 (+ 280us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:18:21.940899 (+ 437us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:18:21.941028 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:21.941087 (+ 59us) server_negotiation.cc:299] Negotiation successful
0618 22:18:21.941139 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260618 22:18:21.944072 7574 raft_consensus.cc:1261] T cce659151b544b58affff848b40339ca P d7dc899699d84381a27dbdf8dae05cc2 [term 1 FOLLOWER]: Refusing update from remote peer 5256bda72de544e985aaad69508c7748: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:18:21.944391 7437 raft_consensus.cc:1261] T cce659151b544b58affff848b40339ca P 4c35cb6b49cd461b83b7ea8f6e4dcb54 [term 1 FOLLOWER]: Refusing update from remote peer 5256bda72de544e985aaad69508c7748: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:18:21.944618 7870 consensus_queue.cc:1035] T cce659151b544b58affff848b40339ca P 5256bda72de544e985aaad69508c7748 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7dc899699d84381a27dbdf8dae05cc2" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 41835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:18:21.944962 7868 consensus_queue.cc:1035] T cce659151b544b58affff848b40339ca P 5256bda72de544e985aaad69508c7748 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c35cb6b49cd461b83b7ea8f6e4dcb54" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:18:21.946717 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:21.946433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:42267 (local address 127.0.98.66:44617)
0618 22:18:21.946568 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:21.946638 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:42267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260618 22:18:21.948172 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:21.947691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:42267 (local address 127.0.98.65:40773)
0618 22:18:21.948061 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:21.948113 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:42267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":308}
I20260618 22:18:21.951411 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:21.950719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:42267 (local address 127.0.98.66:41397)
0618 22:18:21.951282 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:21.951338 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:42267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":494}
I20260618 22:18:21.951458 7841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:21.951147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:42267 (local address 127.0.98.65:43037)
0618 22:18:21.951318 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:21.951387 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:42267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260618 22:18:21.951951 7869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:21.951114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:42267 (local address 127.0.98.67:51551)
0618 22:18:21.951825 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:21.951882 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:42267: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":641}
I20260618 22:18:21.959000 7295 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:18:21.959123 7295 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36474:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:18:21.962725 7295 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9d8bc0e14290456db37208ee7d35b1cc: Sending DeleteTablet for 3 replicas of tablet cce659151b544b58affff848b40339ca
I20260618 22:18:21.963454 7690 tablet_service.cc:1514] Processing DeleteTablet for tablet cce659151b544b58affff848b40339ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:18:21 UTC) from {username='oryx', principal='oryx/127.0.98.124@KRBTEST.COM'} at 127.0.0.1:41298
I20260618 22:18:21.963680 7417 tablet_service.cc:1514] Processing DeleteTablet for tablet cce659151b544b58affff848b40339ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:18:21 UTC) from {username='oryx', principal='oryx/127.0.98.124@KRBTEST.COM'} at 127.0.0.1:37682
I20260618 22:18:21.963742 7554 tablet_service.cc:1514] Processing DeleteTablet for tablet cce659151b544b58affff848b40339ca with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:18:21 UTC) from {username='oryx', principal='oryx/127.0.98.124@KRBTEST.COM'} at 127.0.0.1:51858
I20260618 22:18:21.964063 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 7259
I20260618 22:18:21.964643 7883 tablet_replica.cc:330] T cce659151b544b58affff848b40339ca P 5256bda72de544e985aaad69508c7748: stopping tablet replica
I20260618 22:18:21.964774 7883 raft_consensus.cc:2229] T cce659151b544b58affff848b40339ca P 5256bda72de544e985aaad69508c7748 [term 1 LEADER]: Raft consensus shutting down.
I20260618 22:18:21.965072 7883 raft_consensus.cc:2258] T cce659151b544b58affff848b40339ca P 5256bda72de544e985aaad69508c7748 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:18:21.966671 7883 ts_tablet_manager.cc:1905] T cce659151b544b58affff848b40339ca P 5256bda72de544e985aaad69508c7748: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:18:21.970088 7883 ts_tablet_manager.cc:1918] T cce659151b544b58affff848b40339ca P 5256bda72de544e985aaad69508c7748: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:18:21.970117 7885 tablet_replica.cc:330] T cce659151b544b58affff848b40339ca P d7dc899699d84381a27dbdf8dae05cc2: stopping tablet replica
I20260618 22:18:21.970247 7883 log.cc:1196] T cce659151b544b58affff848b40339ca P 5256bda72de544e985aaad69508c7748: Deleting WAL directory at /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781821065380026-393-0/minicluster-data/ts-2/wal/wals/cce659151b544b58affff848b40339ca
I20260618 22:18:21.970252 7885 raft_consensus.cc:2229] T cce659151b544b58affff848b40339ca P d7dc899699d84381a27dbdf8dae05cc2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:18:21.970373 7885 raft_consensus.cc:2258] T cce659151b544b58affff848b40339ca P d7dc899699d84381a27dbdf8dae05cc2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:18:21.970677 7883 ts_tablet_manager.cc:1939] T cce659151b544b58affff848b40339ca P 5256bda72de544e985aaad69508c7748: Deleting consensus metadata
I20260618 22:18:21.971096 7885 ts_tablet_manager.cc:1905] T cce659151b544b58affff848b40339ca P d7dc899699d84381a27dbdf8dae05cc2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:18:21.971498 7290 catalog_manager.cc:4841] TS 5256bda72de544e985aaad69508c7748 (127.0.98.67:46203): tablet cce659151b544b58affff848b40339ca (table test-table [id=5a3d33af4e404641ac3bec5eaab839ed]) successfully deleted
I20260618 22:18:21.973037 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 7491
I20260618 22:18:21.973443 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:21.973084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:34925 (local address 127.0.0.1:37696)
0618 22:18:21.973281 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:21.973351 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.65:34925: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
W20260618 22:18:21.974928 7291 catalog_manager.cc:4584] TS 4c35cb6b49cd461b83b7ea8f6e4dcb54 (127.0.98.65:34925): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cce659151b544b58affff848b40339ca: Network error: Client connection negotiation failed: client connection to 127.0.98.65:34925: connect: Connection refused (error 111)
I20260618 22:18:21.979122 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 7627
I20260618 22:18:21.979408 7855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:21.979118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:41835 (local address 127.0.0.1:51864)
0618 22:18:21.979261 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:21.979325 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.66:41835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260618 22:18:21.984190 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 6978
I20260618 22:18:21.989637 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 7087
2026-06-18T22:18:21Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7296 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:18:22 dist-test-slave-g6r4 krb5kdc[7888](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:18:22 dist-test-slave-g6r4 krb5kdc[7888](info): set up 2 sockets
Jun 18 22:18:22 dist-test-slave-g6r4 krb5kdc[7888](info): commencing operation
krb5kdc: starting...
W20260618 22:18:23.022411 393 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/krb5kdc/test-user.keytab.
Jun 18 22:18:23 dist-test-slave-g6r4 krb5kdc[7888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:18:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:18:23Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/krb5kdc/oryx_127.0.98.126.keytab.
Entry for principal oryx/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/krb5kdc/oryx_127.0.98.126.keytab.
I20260618 22:18:23.138862 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:37239
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/krb5kdc/oryx_127.0.98.126.keytab
--principal=oryx/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:34537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:37239
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:18:23.244660 7901 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:18:23.244913 7901 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:18:23.244953 7901 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:18:23.248904 7901 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:18:23.248957 7901 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260618 22:18:23.248975 7901 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:18:23.248994 7901 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:18:23.249009 7901 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:18:23.249070 7901 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:18:23.254307 7901 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:34537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:37239
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/krb5kdc/oryx_127.0.98.126.keytab
--principal=oryx/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:37239
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7901
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:18:23.255263 7901 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:18:23.256251 7901 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:18:23.265199 7909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:23.265196 7906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:23.265232 7907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:23.265233 7901 server_base.cc:1034] running on GCE node
I20260618 22:18:23.266001 7901 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:18:23.267015 7901 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:18:23.268167 7901 hybrid_clock.cc:648] HybridClock initialized: now 1781821103268141 us; error 43 us; skew 500 ppm
Jun 18 22:18:23 dist-test-slave-g6r4 krb5kdc[7888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821103, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:18:23.270884 7901 init.cc:377] Logged in from keytab as oryx/127.0.98.126@KRBTEST.COM (short username oryx)
I20260618 22:18:23.271740 7901 webserver.cc:460] Webserver started at http://127.0.98.126:42053/ using document root <none> and password file <none>
I20260618 22:18:23.272262 7901 fs_manager.cc:359] Metadata directory not provided
I20260618 22:18:23.272321 7901 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:18:23.272552 7901 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:18:23.274562 7901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/master-0/data/instance:
uuid: "ec613fe2b9494be494f0f98c8cfe0d63"
format_stamp: "Formatted at 2026-06-18 22:18:23 on dist-test-slave-g6r4"
server_key: "6505b47e81b33f9d7d6f2ffe3750d3fa"
server_key_iv: "15426567a09ce302a7dc2c4c66f2e914"
server_key_version: "encryptionkey@0"
I20260618 22:18:23.275091 7901 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/master-0/wal/instance:
uuid: "ec613fe2b9494be494f0f98c8cfe0d63"
format_stamp: "Formatted at 2026-06-18 22:18:23 on dist-test-slave-g6r4"
server_key: "6505b47e81b33f9d7d6f2ffe3750d3fa"
server_key_iv: "15426567a09ce302a7dc2c4c66f2e914"
server_key_version: "encryptionkey@0"
I20260618 22:18:23.278559 7901 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:18:23.280826 7916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:23.282047 7901 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:18:23.282174 7901 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/master-0/wal
uuid: "ec613fe2b9494be494f0f98c8cfe0d63"
format_stamp: "Formatted at 2026-06-18 22:18:23 on dist-test-slave-g6r4"
server_key: "6505b47e81b33f9d7d6f2ffe3750d3fa"
server_key_iv: "15426567a09ce302a7dc2c4c66f2e914"
server_key_version: "encryptionkey@0"
I20260618 22:18:23.282276 7901 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:18:23.297178 7901 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:18:23.300053 7901 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:18:23.300254 7901 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:18:23.308149 7901 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:37239
I20260618 22:18:23.308157 7978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:37239 every 8 connection(s)
I20260618 22:18:23.309124 7901 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
I20260618 22:18:23.311748 7979 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:18:23.314297 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 7901
I20260618 22:18:23.314426 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/master-0/wal/instance
I20260618 22:18:23.314723 393 external_mini_cluster.cc:1428] Setting key 4f2f9e54ab9915b7574505d41d7af9d0
I20260618 22:18:23.317580 7979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec613fe2b9494be494f0f98c8cfe0d63: Bootstrap starting.
I20260618 22:18:23.320006 7979 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec613fe2b9494be494f0f98c8cfe0d63: Neither blocks nor log segments found. Creating new log.
I20260618 22:18:23.320850 7979 log.cc:824] T 00000000000000000000000000000000 P ec613fe2b9494be494f0f98c8cfe0d63: Log is configured to *not* fsync() on all Append() calls
I20260618 22:18:23.322595 7979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec613fe2b9494be494f0f98c8cfe0d63: No bootstrap required, opened a new log
Jun 18 22:18:23 dist-test-slave-g6r4 krb5kdc[7888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM
I20260618 22:18:23.325737 7979 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec613fe2b9494be494f0f98c8cfe0d63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec613fe2b9494be494f0f98c8cfe0d63" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 37239 } }
I20260618 22:18:23.325987 7979 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec613fe2b9494be494f0f98c8cfe0d63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:18:23.326049 7979 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec613fe2b9494be494f0f98c8cfe0d63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec613fe2b9494be494f0f98c8cfe0d63, State: Initialized, Role: FOLLOWER
I20260618 22:18:23.326521 7979 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec613fe2b9494be494f0f98c8cfe0d63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec613fe2b9494be494f0f98c8cfe0d63" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 37239 } }
I20260618 22:18:23.326645 7979 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec613fe2b9494be494f0f98c8cfe0d63 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:18:23.326735 7979 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec613fe2b9494be494f0f98c8cfe0d63 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:18:23.326848 7979 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec613fe2b9494be494f0f98c8cfe0d63 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:18:23.327693 7979 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec613fe2b9494be494f0f98c8cfe0d63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec613fe2b9494be494f0f98c8cfe0d63" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 37239 } }
I20260618 22:18:23.328049 7979 leader_election.cc:302] T 00000000000000000000000000000000 P ec613fe2b9494be494f0f98c8cfe0d63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec613fe2b9494be494f0f98c8cfe0d63; no voters:
I20260618 22:18:23.328336 7979 leader_election.cc:288] T 00000000000000000000000000000000 P ec613fe2b9494be494f0f98c8cfe0d63 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:18:23.328438 7984 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec613fe2b9494be494f0f98c8cfe0d63 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:18:23.328647 7984 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec613fe2b9494be494f0f98c8cfe0d63 [term 1 LEADER]: Becoming Leader. State: Replica: ec613fe2b9494be494f0f98c8cfe0d63, State: Running, Role: LEADER
I20260618 22:18:23.328760 7982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:23.315994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55926 (local address 127.0.98.126:37239)
0618 22:18:23.316372 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:23.316397 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:23.316430 (+ 33us) server_negotiation.cc:407] Connection header received
0618 22:18:23.317052 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:23.317068 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:23.317380 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:23.317644 (+ 264us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:23.318558 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:23.319524 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:23.320685 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:23.321195 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:23.321368 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:23.323872 (+ 2504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:23.323877 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:23.323889 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:23.323913 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:23.325566 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:23.326026 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:23.326034 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:23.326040 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:23.326209 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:23.326502 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:23.326506 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:23.326508 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:23.327041 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:23.327245 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:23.327527 (+ 282us) server_negotiation.cc:299] Negotiation successful
0618 22:18:23.327826 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":124,"threads_started":1}
I20260618 22:18:23.329015 7984 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec613fe2b9494be494f0f98c8cfe0d63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec613fe2b9494be494f0f98c8cfe0d63" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 37239 } }
I20260618 22:18:23.329491 7979 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec613fe2b9494be494f0f98c8cfe0d63 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:18:23.331560 7985 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec613fe2b9494be494f0f98c8cfe0d63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec613fe2b9494be494f0f98c8cfe0d63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec613fe2b9494be494f0f98c8cfe0d63" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 37239 } } }
I20260618 22:18:23.331732 7986 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec613fe2b9494be494f0f98c8cfe0d63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec613fe2b9494be494f0f98c8cfe0d63. Latest consensus state: current_term: 1 leader_uuid: "ec613fe2b9494be494f0f98c8cfe0d63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec613fe2b9494be494f0f98c8cfe0d63" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 37239 } } }
I20260618 22:18:23.332006 7985 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec613fe2b9494be494f0f98c8cfe0d63 [sys.catalog]: This master's current role is: LEADER
I20260618 22:18:23.332010 7986 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec613fe2b9494be494f0f98c8cfe0d63 [sys.catalog]: This master's current role is: LEADER
W20260618 22:18:23.334292 7998 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ec613fe2b9494be494f0f98c8cfe0d63: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 22:18:23.334367 7998 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 22:18:23.334421 7999 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:18:23.335273 7999 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:18:23.340348 7999 catalog_manager.cc:1347] Generated new cluster ID: 46917b023938484fa8b4afaaccaea742
I20260618 22:18:23.340407 7999 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:18:23.361826 7999 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:18:23.362748 7999 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:18:23.366705 7999 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec613fe2b9494be494f0f98c8cfe0d63: Generated new TSK 0
I20260618 22:18:23.367321 7999 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/krb5kdc/oryx_127.0.98.65.keytab.
Entry for principal oryx/127.0.98.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/krb5kdc/oryx_127.0.98.65.keytab.
I20260618 22:18:23.396595 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:37239
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/krb5kdc/oryx_127.0.98.65.keytab
--principal=oryx/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:34537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:18:23.520650 8005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:18:23.520885 8005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:18:23.520951 8005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:18:23.524983 8005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:18:23.525055 8005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:18:23.525095 8005 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260618 22:18:23.525126 8005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260618 22:18:23.530267 8005 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:34537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/krb5kdc/oryx_127.0.98.65.keytab
--principal=oryx/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.126:37239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8005
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:18:23.531294 8005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:18:23.532315 8005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:18:23.541556 8013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:23.541692 8005 server_base.cc:1034] running on GCE node
W20260618 22:18:23.541520 8011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:23.541556 8010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:23.542228 8005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:18:23.542822 8005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:18:23.544003 8005 hybrid_clock.cc:648] HybridClock initialized: now 1781821103543980 us; error 47 us; skew 500 ppm
Jun 18 22:18:23 dist-test-slave-g6r4 krb5kdc[7888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821103, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:18:23.546752 8005 init.cc:377] Logged in from keytab as oryx/127.0.98.65@KRBTEST.COM (short username oryx)
I20260618 22:18:23.547511 8005 webserver.cc:460] Webserver started at http://127.0.98.65:35871/ using document root <none> and password file <none>
I20260618 22:18:23.547989 8005 fs_manager.cc:359] Metadata directory not provided
I20260618 22:18:23.548044 8005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:18:23.548193 8005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:18:23.550102 8005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/ts-0/data/instance:
uuid: "f85c3f5cab7b4442b6f829cfbadd33b5"
format_stamp: "Formatted at 2026-06-18 22:18:23 on dist-test-slave-g6r4"
server_key: "b4185ccf7f04902cfb33bfe0268e51cd"
server_key_iv: "8734316056f0506924071f3556a28d98"
server_key_version: "encryptionkey@0"
I20260618 22:18:23.550602 8005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/ts-0/wal/instance:
uuid: "f85c3f5cab7b4442b6f829cfbadd33b5"
format_stamp: "Formatted at 2026-06-18 22:18:23 on dist-test-slave-g6r4"
server_key: "b4185ccf7f04902cfb33bfe0268e51cd"
server_key_iv: "8734316056f0506924071f3556a28d98"
server_key_version: "encryptionkey@0"
I20260618 22:18:23.553997 8005 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260618 22:18:23.556254 8020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:23.557396 8005 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:18:23.557490 8005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/ts-0/wal
uuid: "f85c3f5cab7b4442b6f829cfbadd33b5"
format_stamp: "Formatted at 2026-06-18 22:18:23 on dist-test-slave-g6r4"
server_key: "b4185ccf7f04902cfb33bfe0268e51cd"
server_key_iv: "8734316056f0506924071f3556a28d98"
server_key_version: "encryptionkey@0"
I20260618 22:18:23.557569 8005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:18:23.568565 8005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:18:23.571565 8005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:18:23.571700 8005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:18:23.572984 8005 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:18:23.573058 8005 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:23.573125 8005 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:18:23.573175 8005 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 18 22:18:23 dist-test-slave-g6r4 krb5kdc[7888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821103, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM
I20260618 22:18:23.584456 8005 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:39557
I20260618 22:18:23.584813 8135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:39557 every 8 connection(s)
I20260618 22:18:23.585613 8005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
I20260618 22:18:23.587571 7982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:23.573732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:38445 (local address 127.0.98.126:37239)
0618 22:18:23.573853 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:23.573856 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:23.574854 (+ 998us) server_negotiation.cc:407] Connection header received
0618 22:18:23.575886 (+ 1032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:23.575891 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:23.575954 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:23.576015 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:23.577696 (+ 1681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:23.578356 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:23.579616 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:23.579905 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:23.580027 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:23.583722 (+ 3695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:23.583725 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:23.583728 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:23.583755 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:23.585499 (+ 1744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:23.586110 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:23.586117 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:23.586119 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:23.586224 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:23.586623 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:23.586626 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:23.586627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:23.586853 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:23.586930 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:23.587361 (+ 431us) server_negotiation.cc:299] Negotiation successful
0618 22:18:23.587458 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260618 22:18:23.588349 8029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:23.574270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37239 (local address 127.0.98.65:38445)
0618 22:18:23.574720 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:23.574757 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:23.575637 (+ 880us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:23.576480 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:23.576490 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:23.576933 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:23.577543 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:23.577578 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:23.578481 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:23.578487 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:23.579479 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:23.579489 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:23.580067 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:23.580070 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:23.580291 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:23.580945 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:23.580968 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:23.583557 (+ 2589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:18:23.585621 (+ 2064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:23.585625 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:23.585635 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:23.585978 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:23.586335 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:23.586338 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:23.586340 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:23.586521 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:23.586941 (+ 420us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:18:23.586945 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:18:23.587140 (+ 195us) client_negotiation.cc:769] Sending connection context
0618 22:18:23.587285 (+ 145us) client_negotiation.cc:240] Negotiation successful
0618 22:18:23.587499 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":129,"threads_started":1}
I20260618 22:18:23.589566 8136 heartbeater.cc:347] Connected to a master server at 127.0.98.126:37239
I20260618 22:18:23.589699 8136 heartbeater.cc:464] Registering TS with master...
I20260618 22:18:23.590210 8136 heartbeater.cc:511] Master 127.0.98.126:37239 requested a full tablet report, sending...
I20260618 22:18:23.591645 7932 ts_manager.cc:194] Registered new tserver with Master: f85c3f5cab7b4442b6f829cfbadd33b5 (127.0.98.65:39557)
I20260618 22:18:23.592767 7932 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.65@KRBTEST.COM'} at 127.0.98.65:38445
I20260618 22:18:23.593103 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 8005
I20260618 22:18:23.593235 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/ts-0/wal/instance
I20260618 22:18:23.593571 393 external_mini_cluster.cc:1428] Setting key 9e3276e5552eba06d11995ca0ca47be7
I20260618 22:18:23.600011 7982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:23.591638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:38443 (local address 127.0.98.126:37239)
0618 22:18:23.593430 (+ 1792us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:23.593434 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:23.593450 (+ 16us) server_negotiation.cc:407] Connection header received
0618 22:18:23.593482 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:23.593485 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:23.593526 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:23.593607 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:23.594230 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:23.594958 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:23.595743 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:23.595959 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:23.596067 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:23.597040 (+ 973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:23.597044 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:23.597046 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:23.597075 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:23.598415 (+ 1340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:23.598905 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:23.598912 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:23.598913 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:23.598979 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:23.599256 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:23.599259 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:23.599261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:23.599418 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:23.599519 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:23.599774 (+ 255us) server_negotiation.cc:299] Negotiation successful
0618 22:18:23.599878 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1717}
I20260618 22:18:23.600023 8142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:23.591527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37239 (local address 127.0.98.65:38443)
0618 22:18:23.591810 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:23.591826 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:23.591907 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:23.593638 (+ 1731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:23.593641 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:23.593845 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:23.593975 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:23.593983 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:23.595095 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:23.595097 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:23.595622 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:23.595629 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:23.596067 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:23.596070 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:23.596167 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:23.596617 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:23.596637 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:23.596910 (+ 273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:18:23.598519 (+ 1609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:23.598525 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:23.598528 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:23.598800 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:23.599077 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:23.599080 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:23.599082 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:23.599163 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:23.599519 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:18:23.599523 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:18:23.599611 (+ 88us) client_negotiation.cc:769] Sending connection context
0618 22:18:23.599724 (+ 113us) client_negotiation.cc:240] Negotiation successful
0618 22:18:23.599854 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":89,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/krb5kdc/oryx_127.0.98.66.keytab.
Entry for principal oryx/127.0.98.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/krb5kdc/oryx_127.0.98.66.keytab.
I20260618 22:18:23.618999 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:37239
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/krb5kdc/oryx_127.0.98.66.keytab
--principal=oryx/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:34537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:18:23.725122 8145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:18:23.725355 8145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:18:23.725397 8145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:18:23.729455 8145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:18:23.729504 8145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:18:23.729542 8145 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260618 22:18:23.729602 8145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260618 22:18:23.734758 8145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:34537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/krb5kdc/oryx_127.0.98.66.keytab
--principal=oryx/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.126:37239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:18:23.735680 8145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:18:23.736631 8145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:18:23.745796 8150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:23.745791 8151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:23.745934 8145 server_base.cc:1034] running on GCE node
W20260618 22:18:23.745811 8153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:23.746512 8145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:18:23.747105 8145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:18:23.748260 8145 hybrid_clock.cc:648] HybridClock initialized: now 1781821103748230 us; error 52 us; skew 500 ppm
Jun 18 22:18:23 dist-test-slave-g6r4 krb5kdc[7888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821103, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:18:23.750887 8145 init.cc:377] Logged in from keytab as oryx/127.0.98.66@KRBTEST.COM (short username oryx)
I20260618 22:18:23.751711 8145 webserver.cc:460] Webserver started at http://127.0.98.66:46485/ using document root <none> and password file <none>
I20260618 22:18:23.752272 8145 fs_manager.cc:359] Metadata directory not provided
I20260618 22:18:23.752362 8145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:18:23.752578 8145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:18:23.754552 8145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/ts-1/data/instance:
uuid: "e94305e8231b4fd9ad7f29754b581d84"
format_stamp: "Formatted at 2026-06-18 22:18:23 on dist-test-slave-g6r4"
server_key: "2fb8fc94e6837d1d58f7ffbd49e83133"
server_key_iv: "d92679ae565b7f05cb9354cf364beed8"
server_key_version: "encryptionkey@0"
I20260618 22:18:23.755100 8145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/ts-1/wal/instance:
uuid: "e94305e8231b4fd9ad7f29754b581d84"
format_stamp: "Formatted at 2026-06-18 22:18:23 on dist-test-slave-g6r4"
server_key: "2fb8fc94e6837d1d58f7ffbd49e83133"
server_key_iv: "d92679ae565b7f05cb9354cf364beed8"
server_key_version: "encryptionkey@0"
I20260618 22:18:23.758594 8145 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260618 22:18:23.760924 8160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:23.762256 8145 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260618 22:18:23.762384 8145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/ts-1/wal
uuid: "e94305e8231b4fd9ad7f29754b581d84"
format_stamp: "Formatted at 2026-06-18 22:18:23 on dist-test-slave-g6r4"
server_key: "2fb8fc94e6837d1d58f7ffbd49e83133"
server_key_iv: "d92679ae565b7f05cb9354cf364beed8"
server_key_version: "encryptionkey@0"
I20260618 22:18:23.762485 8145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:18:23.794480 8145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:18:23.797497 8145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:18:23.797716 8145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:18:23.799291 8145 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:18:23.799368 8145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:23.799427 8145 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:18:23.799481 8145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 18 22:18:23 dist-test-slave-g6r4 krb5kdc[7888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821103, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM
I20260618 22:18:23.809921 8145 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:39465
I20260618 22:18:23.810050 8275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:39465 every 8 connection(s)
I20260618 22:18:23.811079 8145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
I20260618 22:18:23.813167 7982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:23.800153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:46367 (local address 127.0.98.126:37239)
0618 22:18:23.800376 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:23.800382 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:23.801149 (+ 767us) server_negotiation.cc:407] Connection header received
0618 22:18:23.802312 (+ 1163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:23.802318 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:23.802391 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:23.802454 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:23.803837 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:23.804764 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:23.806427 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:23.806648 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:23.806765 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:23.809358 (+ 2593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:23.809362 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:23.809365 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:23.809397 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:23.811104 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:23.811687 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:23.811690 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:23.811692 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:23.811779 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:23.812163 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:23.812166 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:23.812168 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:23.812356 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:23.812437 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:23.812955 (+ 518us) server_negotiation.cc:299] Negotiation successful
0618 22:18:23.813052 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260618 22:18:23.814026 8169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:23.800542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37239 (local address 127.0.98.66:46367)
0618 22:18:23.800979 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:23.801026 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:23.802076 (+ 1050us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:23.802848 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:23.802857 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:23.803240 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:23.803704 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:23.803717 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:23.804906 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:23.804910 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:23.806292 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:23.806301 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:23.806808 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:23.806812 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:23.806989 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:23.807568 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:23.807584 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:23.809185 (+ 1601us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:18:23.811235 (+ 2050us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:23.811241 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:23.811255 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:23.811580 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:23.811887 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:23.811893 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:23.811895 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:23.812054 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:23.812451 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:18:23.812456 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:18:23.812750 (+ 294us) client_negotiation.cc:769] Sending connection context
0618 22:18:23.812902 (+ 152us) client_negotiation.cc:240] Negotiation successful
0618 22:18:23.813130 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":130,"threads_started":1}
I20260618 22:18:23.814800 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 8145
I20260618 22:18:23.814965 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/ts-1/wal/instance
I20260618 22:18:23.815223 8276 heartbeater.cc:347] Connected to a master server at 127.0.98.126:37239
I20260618 22:18:23.815312 393 external_mini_cluster.cc:1428] Setting key 0592d6becca9573772ddd59763c21b19
I20260618 22:18:23.815361 8276 heartbeater.cc:464] Registering TS with master...
I20260618 22:18:23.815883 8276 heartbeater.cc:511] Master 127.0.98.126:37239 requested a full tablet report, sending...
I20260618 22:18:23.816865 7933 ts_manager.cc:194] Registered new tserver with Master: e94305e8231b4fd9ad7f29754b581d84 (127.0.98.66:39465)
I20260618 22:18:23.817590 7933 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.66@KRBTEST.COM'} at 127.0.98.66:46367
WARNING: no policy specified for oryx/127.0.98.67@KRBTEST.COM; defaulting to no policy
I20260618 22:18:23.825497 7982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:23.817479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:38843 (local address 127.0.98.126:37239)
0618 22:18:23.817596 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:23.817601 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:23.818720 (+ 1119us) server_negotiation.cc:407] Connection header received
0618 22:18:23.819026 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:23.819031 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:23.819086 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:23.819173 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:23.819629 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:23.820146 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:23.820987 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:23.821198 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:23.821302 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:23.822396 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:23.822400 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:23.822402 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:23.822430 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:23.823833 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:23.824256 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:23.824260 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:23.824262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:23.824379 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:23.824658 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:23.824663 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:23.824665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:23.824864 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:23.824969 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:23.825233 (+ 264us) server_negotiation.cc:299] Negotiation successful
0618 22:18:23.825342 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260618 22:18:23.825515 8282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:23.817562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37239 (local address 127.0.98.66:38843)
0618 22:18:23.818586 (+ 1024us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:23.818601 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:23.818923 (+ 322us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:23.819203 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:23.819206 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:23.819410 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:23.819514 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:23.819521 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:23.820274 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:23.820278 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:23.820857 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:23.820865 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:23.821305 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:23.821309 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:23.821399 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:23.821930 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:23.821945 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:23.822258 (+ 313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:18:23.823961 (+ 1703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:23.823965 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:23.823967 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:23.824158 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:23.824480 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:23.824483 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:23.824485 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:23.824569 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:23.824970 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:18:23.824973 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:18:23.825064 (+ 91us) client_negotiation.cc:769] Sending connection context
0618 22:18:23.825167 (+ 103us) client_negotiation.cc:240] Negotiation successful
0618 22:18:23.825339 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":950,"thread_start_us":92,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/krb5kdc/oryx_127.0.98.67.keytab.
Entry for principal oryx/127.0.98.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/krb5kdc/oryx_127.0.98.67.keytab.
I20260618 22:18:23.842362 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:37239
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/krb5kdc/oryx_127.0.98.67.keytab
--principal=oryx/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:34537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:18:23.949190 8285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:18:23.949424 8285 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:18:23.949466 8285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:18:23.953382 8285 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:18:23.953431 8285 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:18:23.953469 8285 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260618 22:18:23.953501 8285 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260618 22:18:23.958622 8285 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:34537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/krb5kdc/oryx_127.0.98.67.keytab
--principal=oryx/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.126:37239
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8285
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:18:23.959506 8285 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:18:23.960417 8285 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:18:23.970767 8291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:23.970836 8290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:23.970769 8293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:23.971101 8285 server_base.cc:1034] running on GCE node
I20260618 22:18:23.971649 8285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:18:23.972245 8285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:18:23.973484 8285 hybrid_clock.cc:648] HybridClock initialized: now 1781821103973450 us; error 47 us; skew 500 ppm
Jun 18 22:18:23 dist-test-slave-g6r4 krb5kdc[7888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821103, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:18:23.976475 8285 init.cc:377] Logged in from keytab as oryx/127.0.98.67@KRBTEST.COM (short username oryx)
I20260618 22:18:23.977401 8285 webserver.cc:460] Webserver started at http://127.0.98.67:40175/ using document root <none> and password file <none>
I20260618 22:18:23.978065 8285 fs_manager.cc:359] Metadata directory not provided
I20260618 22:18:23.978161 8285 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:18:23.978397 8285 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:18:23.980502 8285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/ts-2/data/instance:
uuid: "2fb18f58d30d4e69998fd6b5bd106bff"
format_stamp: "Formatted at 2026-06-18 22:18:23 on dist-test-slave-g6r4"
server_key: "92b77d01926045ee5e6a9724c81cef3e"
server_key_iv: "64c7b774269cf8a6f7c680f61253d8d1"
server_key_version: "encryptionkey@0"
I20260618 22:18:23.981122 8285 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/ts-2/wal/instance:
uuid: "2fb18f58d30d4e69998fd6b5bd106bff"
format_stamp: "Formatted at 2026-06-18 22:18:23 on dist-test-slave-g6r4"
server_key: "92b77d01926045ee5e6a9724c81cef3e"
server_key_iv: "64c7b774269cf8a6f7c680f61253d8d1"
server_key_version: "encryptionkey@0"
I20260618 22:18:23.985056 8285 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260618 22:18:23.987643 8300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:23.988873 8285 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260618 22:18:23.989135 8285 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/ts-2/wal
uuid: "2fb18f58d30d4e69998fd6b5bd106bff"
format_stamp: "Formatted at 2026-06-18 22:18:23 on dist-test-slave-g6r4"
server_key: "92b77d01926045ee5e6a9724c81cef3e"
server_key_iv: "64c7b774269cf8a6f7c680f61253d8d1"
server_key_version: "encryptionkey@0"
I20260618 22:18:23.989285 8285 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:18:24.002002 8285 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:18:24.004681 8285 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:18:24.004878 8285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:18:24.006381 8285 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:18:24.006467 8285 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:24.006521 8285 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:18:24.006554 8285 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 18 22:18:24 dist-test-slave-g6r4 krb5kdc[7888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821103, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM
I20260618 22:18:24.018244 8285 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:39501
I20260618 22:18:24.018851 8415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:39501 every 8 connection(s)
I20260618 22:18:24.019327 8285 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
I20260618 22:18:24.021054 7982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:24.006957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:56061 (local address 127.0.98.126:37239)
0618 22:18:24.007077 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:24.007081 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:24.008117 (+ 1036us) server_negotiation.cc:407] Connection header received
0618 22:18:24.009008 (+ 891us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:24.009012 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:24.009071 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:24.009173 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:24.010485 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.011040 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.013065 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.013381 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.013499 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:24.016864 (+ 3365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:24.016868 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:24.016871 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:24.016897 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:24.018535 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:24.019203 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:24.019207 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:24.019211 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:24.019333 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:24.019787 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:24.019793 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:24.019796 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:24.020070 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:24.020183 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:24.020744 (+ 561us) server_negotiation.cc:299] Negotiation successful
0618 22:18:24.020872 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260618 22:18:24.021780 8309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:24.007503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37239 (local address 127.0.98.67:56061)
0618 22:18:24.007977 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:24.008015 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:24.008811 (+ 796us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:24.009349 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:24.009376 (+ 27us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:24.009847 (+ 471us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:24.010339 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:24.010353 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.011204 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.011209 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:24.012904 (+ 1695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:24.012918 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.013545 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.013549 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:24.013714 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:24.014433 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:24.014451 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:24.016698 (+ 2247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:18:24.018643 (+ 1945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:24.018649 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:24.018662 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:24.019109 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:24.019453 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:24.019456 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:24.019458 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:24.019629 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:24.020183 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:18:24.020191 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:18:24.020492 (+ 301us) client_negotiation.cc:769] Sending connection context
0618 22:18:24.020688 (+ 196us) client_negotiation.cc:240] Negotiation successful
0618 22:18:24.020999 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":149,"threads_started":1}
I20260618 22:18:24.022930 8416 heartbeater.cc:347] Connected to a master server at 127.0.98.126:37239
I20260618 22:18:24.023058 8416 heartbeater.cc:464] Registering TS with master...
I20260618 22:18:24.023576 8416 heartbeater.cc:511] Master 127.0.98.126:37239 requested a full tablet report, sending...
I20260618 22:18:24.024593 7932 ts_manager.cc:194] Registered new tserver with Master: 2fb18f58d30d4e69998fd6b5bd106bff (127.0.98.67:39501)
I20260618 22:18:24.025413 7932 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.67@KRBTEST.COM'} at 127.0.98.67:56061
I20260618 22:18:24.028302 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 8285
I20260618 22:18:24.028453 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/ts-2/wal/instance
I20260618 22:18:24.028752 393 external_mini_cluster.cc:1428] Setting key b89d572bb84a6fc47440bd0ee236c514
I20260618 22:18:24.030938 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 22:18:24.032271 7982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:24.025150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:42877 (local address 127.0.98.126:37239)
0618 22:18:24.025261 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:24.025266 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:24.025424 (+ 158us) server_negotiation.cc:407] Connection header received
0618 22:18:24.025559 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:24.025562 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:24.025607 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:24.025690 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:24.026225 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.026711 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.027521 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.027736 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.027850 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:24.029030 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:24.029037 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:24.029042 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:24.029068 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:24.030799 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:24.031253 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:24.031255 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:24.031256 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:24.031330 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:24.031614 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:24.031616 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:24.031617 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:24.031761 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:24.031834 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:24.032097 (+ 263us) server_negotiation.cc:299] Negotiation successful
0618 22:18:24.032174 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260618 22:18:24.032536 8422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:24.025088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37239 (local address 127.0.98.67:42877)
0618 22:18:24.025345 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:24.025358 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:24.025463 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:24.025801 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:24.025804 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:24.026024 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:24.026114 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:24.026119 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.026838 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.026841 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:24.027401 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:24.027409 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.027848 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.027851 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:24.027946 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:24.028504 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:24.028523 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:24.028883 (+ 360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:18:24.030918 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:24.030921 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:24.030922 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:24.031153 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:24.031421 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:24.031424 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:24.031427 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:24.031527 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:24.031836 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:18:24.031842 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:18:24.031948 (+ 106us) client_negotiation.cc:769] Sending connection context
0618 22:18:24.032072 (+ 124us) client_negotiation.cc:240] Negotiation successful
0618 22:18:24.032250 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":90,"threads_started":1}
Jun 18 22:18:24 dist-test-slave-g6r4 krb5kdc[7888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821104, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 18 22:18:24 dist-test-slave-g6r4 krb5kdc[7888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:18:24 dist-test-slave-g6r4 krb5kdc[7888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
W20260618 22:18:24.054657 7982 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:18:24.048327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38700 (local address 127.0.98.126:37239)
0618 22:18:24.048479 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:24.048483 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:24.048495 (+ 12us) server_negotiation.cc:407] Connection header received
0618 22:18:24.048528 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:24.048530 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:24.048564 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:24.048634 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:24.049342 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.049862 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.050787 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.050991 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.051069 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:24.054543 (+ 3474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38700: BlockingRecv error: recv got EOF from 127.0.0.1:38700 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 18 22:18:24 dist-test-slave-g6r4 krb5kdc[7888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821104, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM
I20260618 22:18:24.063949 7982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:24.055920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38712 (local address 127.0.98.126:37239)
0618 22:18:24.056024 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:24.056028 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:24.056158 (+ 130us) server_negotiation.cc:407] Connection header received
0618 22:18:24.056300 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:24.056303 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:24.056348 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:24.056431 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:24.057092 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.057579 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.058451 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.058659 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.058785 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:24.061060 (+ 2275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:24.061063 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:24.061065 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:24.061086 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:24.062586 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:24.062972 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:24.062975 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:24.062976 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:24.063048 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:24.063268 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:24.063270 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:24.063271 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:24.063436 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:24.063531 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:24.063756 (+ 225us) server_negotiation.cc:299] Negotiation successful
0618 22:18:24.063839 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260618 22:18:24.066672 7932 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38712:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 22:18:24.069463 7932 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:18:24.083464 8442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:24.078311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:39465 (local address 127.0.0.1:36894)
0618 22:18:24.078970 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:24.078982 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:24.079071 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:24.079494 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:24.079496 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:24.079514 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:18:24.079616 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:24.079621 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.080607 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.080612 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:24.081866 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:24.081875 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.082928 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.082932 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:24.083036 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:24.083072 (+ 36us) client_negotiation.cc:769] Sending connection context
0618 22:18:24.083189 (+ 117us) client_negotiation.cc:240] Negotiation successful
0618 22:18:24.083299 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"mutex_wait_us":70,"thread_start_us":91,"threads_started":1}
I20260618 22:18:24.083791 8439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:24.077677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58350 (local address 127.0.98.65:39557)
0618 22:18:24.078805 (+ 1128us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:24.078819 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:24.078840 (+ 21us) server_negotiation.cc:407] Connection header received
0618 22:18:24.078916 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:24.078922 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:24.079132 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:24.079433 (+ 301us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:18:24.079797 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.080938 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.082654 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.083184 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.083318 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:24.083472 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:24.083567 (+ 95us) server_negotiation.cc:299] Negotiation successful
0618 22:18:24.083656 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1026,"thread_start_us":87,"threads_started":1}
I20260618 22:18:24.084023 8443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:24.078492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36894 (local address 127.0.98.66:39465)
0618 22:18:24.078896 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:24.078917 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:24.079066 (+ 149us) server_negotiation.cc:407] Connection header received
0618 22:18:24.079188 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:24.079195 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:24.079359 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:24.079487 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:18:24.079716 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.080489 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.082004 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.082612 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.082712 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:24.082869 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:24.083815 (+ 946us) server_negotiation.cc:299] Negotiation successful
0618 22:18:24.083899 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":115,"threads_started":1}
I20260618 22:18:24.083746 8440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:24.077518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:39557 (local address 127.0.0.1:58350)
0618 22:18:24.078353 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:24.078401 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:24.078522 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:24.079282 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:24.079285 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:24.079304 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:18:24.079654 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:24.079681 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.081060 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.081063 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:24.082544 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:24.082551 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.083316 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.083319 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:24.083426 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:24.083449 (+ 23us) client_negotiation.cc:769] Sending connection context
0618 22:18:24.083535 (+ 86us) client_negotiation.cc:240] Negotiation successful
0618 22:18:24.083608 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"mutex_wait_us":434,"thread_start_us":74,"threads_started":1}
I20260618 22:18:24.084396 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:24.077729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45698 (local address 127.0.98.67:39501)
0618 22:18:24.078126 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:24.078134 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:24.078786 (+ 652us) server_negotiation.cc:407] Connection header received
0618 22:18:24.078870 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:24.078876 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:24.079089 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:24.079253 (+ 164us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:18:24.079716 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.080927 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.082515 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.083306 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.083399 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:24.083551 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:24.084126 (+ 575us) server_negotiation.cc:299] Negotiation successful
0618 22:18:24.084224 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":97,"threads_started":1}
I20260618 22:18:24.084009 8438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:24.077522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:39501 (local address 127.0.0.1:45698)
0618 22:18:24.078589 (+ 1067us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:24.078602 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:24.078690 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:24.079245 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:24.079250 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:24.079271 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:18:24.079494 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:24.079501 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.081060 (+ 1559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.081063 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:24.082386 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:24.082397 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.083633 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.083637 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:24.083720 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:24.083741 (+ 21us) client_negotiation.cc:769] Sending connection context
0618 22:18:24.083809 (+ 68us) client_negotiation.cc:240] Negotiation successful
0618 22:18:24.083876 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1011,"mutex_wait_us":258,"spinlock_wait_cycles":2944,"thread_start_us":100,"threads_started":1}
I20260618 22:18:24.087389 8210 tablet_service.cc:1467] Processing CreateTablet for tablet 07965a958d654f208b94dc29b9d2c635 (DEFAULT_TABLE table=test-table [id=ad25bd072a4a471892d47750d1cf9e1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:18:24.087389 8070 tablet_service.cc:1467] Processing CreateTablet for tablet 07965a958d654f208b94dc29b9d2c635 (DEFAULT_TABLE table=test-table [id=ad25bd072a4a471892d47750d1cf9e1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:18:24.087692 8350 tablet_service.cc:1467] Processing CreateTablet for tablet 07965a958d654f208b94dc29b9d2c635 (DEFAULT_TABLE table=test-table [id=ad25bd072a4a471892d47750d1cf9e1b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:18:24.088451 8210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07965a958d654f208b94dc29b9d2c635. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:18:24.088451 8070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07965a958d654f208b94dc29b9d2c635. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:18:24.088600 8350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07965a958d654f208b94dc29b9d2c635. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:18:24.095052 8444 tablet_bootstrap.cc:492] T 07965a958d654f208b94dc29b9d2c635 P 2fb18f58d30d4e69998fd6b5bd106bff: Bootstrap starting.
I20260618 22:18:24.096138 8446 tablet_bootstrap.cc:492] T 07965a958d654f208b94dc29b9d2c635 P e94305e8231b4fd9ad7f29754b581d84: Bootstrap starting.
I20260618 22:18:24.097110 8444 tablet_bootstrap.cc:654] T 07965a958d654f208b94dc29b9d2c635 P 2fb18f58d30d4e69998fd6b5bd106bff: Neither blocks nor log segments found. Creating new log.
I20260618 22:18:24.097812 8446 tablet_bootstrap.cc:654] T 07965a958d654f208b94dc29b9d2c635 P e94305e8231b4fd9ad7f29754b581d84: Neither blocks nor log segments found. Creating new log.
I20260618 22:18:24.097865 8444 log.cc:824] T 07965a958d654f208b94dc29b9d2c635 P 2fb18f58d30d4e69998fd6b5bd106bff: Log is configured to *not* fsync() on all Append() calls
I20260618 22:18:24.098670 8446 log.cc:824] T 07965a958d654f208b94dc29b9d2c635 P e94305e8231b4fd9ad7f29754b581d84: Log is configured to *not* fsync() on all Append() calls
I20260618 22:18:24.099208 8445 tablet_bootstrap.cc:492] T 07965a958d654f208b94dc29b9d2c635 P f85c3f5cab7b4442b6f829cfbadd33b5: Bootstrap starting.
I20260618 22:18:24.099841 8444 tablet_bootstrap.cc:492] T 07965a958d654f208b94dc29b9d2c635 P 2fb18f58d30d4e69998fd6b5bd106bff: No bootstrap required, opened a new log
I20260618 22:18:24.100028 8444 ts_tablet_manager.cc:1397] T 07965a958d654f208b94dc29b9d2c635 P 2fb18f58d30d4e69998fd6b5bd106bff: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260618 22:18:24.100551 8446 tablet_bootstrap.cc:492] T 07965a958d654f208b94dc29b9d2c635 P e94305e8231b4fd9ad7f29754b581d84: No bootstrap required, opened a new log
I20260618 22:18:24.100725 8446 ts_tablet_manager.cc:1397] T 07965a958d654f208b94dc29b9d2c635 P e94305e8231b4fd9ad7f29754b581d84: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260618 22:18:24.101338 8445 tablet_bootstrap.cc:654] T 07965a958d654f208b94dc29b9d2c635 P f85c3f5cab7b4442b6f829cfbadd33b5: Neither blocks nor log segments found. Creating new log.
I20260618 22:18:24.102033 8445 log.cc:824] T 07965a958d654f208b94dc29b9d2c635 P f85c3f5cab7b4442b6f829cfbadd33b5: Log is configured to *not* fsync() on all Append() calls
I20260618 22:18:24.103561 8445 tablet_bootstrap.cc:492] T 07965a958d654f208b94dc29b9d2c635 P f85c3f5cab7b4442b6f829cfbadd33b5: No bootstrap required, opened a new log
I20260618 22:18:24.103785 8445 ts_tablet_manager.cc:1397] T 07965a958d654f208b94dc29b9d2c635 P f85c3f5cab7b4442b6f829cfbadd33b5: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260618 22:18:24.103899 8444 raft_consensus.cc:348] T 07965a958d654f208b94dc29b9d2c635 P 2fb18f58d30d4e69998fd6b5bd106bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 39557 } } peers { permanent_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39501 } } peers { permanent_uuid: "e94305e8231b4fd9ad7f29754b581d84" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39465 } }
I20260618 22:18:24.104156 8444 raft_consensus.cc:374] T 07965a958d654f208b94dc29b9d2c635 P 2fb18f58d30d4e69998fd6b5bd106bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:18:24.104247 8444 raft_consensus.cc:729] T 07965a958d654f208b94dc29b9d2c635 P 2fb18f58d30d4e69998fd6b5bd106bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fb18f58d30d4e69998fd6b5bd106bff, State: Initialized, Role: FOLLOWER
I20260618 22:18:24.104434 8446 raft_consensus.cc:348] T 07965a958d654f208b94dc29b9d2c635 P e94305e8231b4fd9ad7f29754b581d84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 39557 } } peers { permanent_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39501 } } peers { permanent_uuid: "e94305e8231b4fd9ad7f29754b581d84" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39465 } }
I20260618 22:18:24.104657 8446 raft_consensus.cc:374] T 07965a958d654f208b94dc29b9d2c635 P e94305e8231b4fd9ad7f29754b581d84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:18:24.104749 8446 raft_consensus.cc:729] T 07965a958d654f208b94dc29b9d2c635 P e94305e8231b4fd9ad7f29754b581d84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e94305e8231b4fd9ad7f29754b581d84, State: Initialized, Role: FOLLOWER
I20260618 22:18:24.104681 8444 consensus_queue.cc:260] T 07965a958d654f208b94dc29b9d2c635 P 2fb18f58d30d4e69998fd6b5bd106bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 39557 } } peers { permanent_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39501 } } peers { permanent_uuid: "e94305e8231b4fd9ad7f29754b581d84" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39465 } }
I20260618 22:18:24.105168 8446 consensus_queue.cc:260] T 07965a958d654f208b94dc29b9d2c635 P e94305e8231b4fd9ad7f29754b581d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 39557 } } peers { permanent_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39501 } } peers { permanent_uuid: "e94305e8231b4fd9ad7f29754b581d84" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39465 } }
I20260618 22:18:24.105644 8444 ts_tablet_manager.cc:1428] T 07965a958d654f208b94dc29b9d2c635 P 2fb18f58d30d4e69998fd6b5bd106bff: Time spent starting tablet: real 0.006s user 0.001s sys 0.005s
I20260618 22:18:24.105813 8276 heartbeater.cc:503] Master 127.0.98.126:37239 was elected leader, sending a full tablet report...
I20260618 22:18:24.106202 8416 heartbeater.cc:503] Master 127.0.98.126:37239 was elected leader, sending a full tablet report...
I20260618 22:18:24.106081 8446 ts_tablet_manager.cc:1428] T 07965a958d654f208b94dc29b9d2c635 P e94305e8231b4fd9ad7f29754b581d84: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260618 22:18:24.107455 8445 raft_consensus.cc:348] T 07965a958d654f208b94dc29b9d2c635 P f85c3f5cab7b4442b6f829cfbadd33b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 39557 } } peers { permanent_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39501 } } peers { permanent_uuid: "e94305e8231b4fd9ad7f29754b581d84" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39465 } }
I20260618 22:18:24.107690 8445 raft_consensus.cc:374] T 07965a958d654f208b94dc29b9d2c635 P f85c3f5cab7b4442b6f829cfbadd33b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:18:24.107786 8445 raft_consensus.cc:729] T 07965a958d654f208b94dc29b9d2c635 P f85c3f5cab7b4442b6f829cfbadd33b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f85c3f5cab7b4442b6f829cfbadd33b5, State: Initialized, Role: FOLLOWER
I20260618 22:18:24.108194 8445 consensus_queue.cc:260] T 07965a958d654f208b94dc29b9d2c635 P f85c3f5cab7b4442b6f829cfbadd33b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 39557 } } peers { permanent_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39501 } } peers { permanent_uuid: "e94305e8231b4fd9ad7f29754b581d84" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39465 } }
I20260618 22:18:24.108955 8136 heartbeater.cc:503] Master 127.0.98.126:37239 was elected leader, sending a full tablet report...
I20260618 22:18:24.109067 8445 ts_tablet_manager.cc:1428] T 07965a958d654f208b94dc29b9d2c635 P f85c3f5cab7b4442b6f829cfbadd33b5: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260618 22:18:24.145455 8452 raft_consensus.cc:482] T 07965a958d654f208b94dc29b9d2c635 P f85c3f5cab7b4442b6f829cfbadd33b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:18:24.145659 8452 raft_consensus.cc:504] T 07965a958d654f208b94dc29b9d2c635 P f85c3f5cab7b4442b6f829cfbadd33b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 39557 } } peers { permanent_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39501 } } peers { permanent_uuid: "e94305e8231b4fd9ad7f29754b581d84" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39465 } }
I20260618 22:18:24.146739 8452 leader_election.cc:288] T 07965a958d654f208b94dc29b9d2c635 P f85c3f5cab7b4442b6f829cfbadd33b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2fb18f58d30d4e69998fd6b5bd106bff (127.0.98.67:39501), e94305e8231b4fd9ad7f29754b581d84 (127.0.98.66:39465)
I20260618 22:18:24.150552 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:24.147104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:48155 (local address 127.0.98.67:39501)
0618 22:18:24.147234 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:24.147238 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:24.147458 (+ 220us) server_negotiation.cc:407] Connection header received
0618 22:18:24.147518 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:24.147524 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:24.147579 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:24.147656 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:18:24.147971 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.148525 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.149578 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.150019 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.150138 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:24.150201 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:24.150371 (+ 170us) server_negotiation.cc:299] Negotiation successful
0618 22:18:24.150434 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260618 22:18:24.150568 8453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:24.146980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:39501 (local address 127.0.98.65:48155)
0618 22:18:24.147256 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:24.147274 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:24.147376 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:24.147707 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:24.147710 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:24.147730 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:18:24.147850 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:24.147856 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.148645 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.148648 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:24.149449 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:24.149457 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.150130 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.150134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:24.150235 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:24.150256 (+ 21us) client_negotiation.cc:769] Sending connection context
0618 22:18:24.150356 (+ 100us) client_negotiation.cc:240] Negotiation successful
0618 22:18:24.150420 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":123,"threads_started":1}
I20260618 22:18:24.150879 8443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:24.147499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:33453 (local address 127.0.98.66:39465)
0618 22:18:24.147636 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:24.147643 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:24.147785 (+ 142us) server_negotiation.cc:407] Connection header received
0618 22:18:24.147928 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:24.147931 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:24.147981 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:24.148056 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:18:24.148332 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.148853 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.149836 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.150304 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.150420 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:24.150475 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:24.150680 (+ 205us) server_negotiation.cc:299] Negotiation successful
0618 22:18:24.150772 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260618 22:18:24.151230 8454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:24.147383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:39465 (local address 127.0.98.65:33453)
0618 22:18:24.147696 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:24.147707 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:24.147817 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:24.148078 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:24.148080 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:24.148095 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:18:24.148232 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:24.148237 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.148958 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.148964 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:24.149722 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:24.149729 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.150418 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.150422 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:24.150519 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:24.150546 (+ 27us) client_negotiation.cc:769] Sending connection context
0618 22:18:24.151009 (+ 463us) client_negotiation.cc:240] Negotiation successful
0618 22:18:24.151089 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":121,"threads_started":1}
I20260618 22:18:24.151508 8370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07965a958d654f208b94dc29b9d2c635" candidate_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" is_pre_election: true
I20260618 22:18:24.151846 8370 raft_consensus.cc:2454] T 07965a958d654f208b94dc29b9d2c635 P 2fb18f58d30d4e69998fd6b5bd106bff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f85c3f5cab7b4442b6f829cfbadd33b5 in term 0.
I20260618 22:18:24.151865 8230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07965a958d654f208b94dc29b9d2c635" candidate_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e94305e8231b4fd9ad7f29754b581d84" is_pre_election: true
I20260618 22:18:24.152101 8230 raft_consensus.cc:2454] T 07965a958d654f208b94dc29b9d2c635 P e94305e8231b4fd9ad7f29754b581d84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f85c3f5cab7b4442b6f829cfbadd33b5 in term 0.
I20260618 22:18:24.152343 8023 leader_election.cc:302] T 07965a958d654f208b94dc29b9d2c635 P f85c3f5cab7b4442b6f829cfbadd33b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fb18f58d30d4e69998fd6b5bd106bff, f85c3f5cab7b4442b6f829cfbadd33b5; no voters:
I20260618 22:18:24.152595 8452 raft_consensus.cc:2781] T 07965a958d654f208b94dc29b9d2c635 P f85c3f5cab7b4442b6f829cfbadd33b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:18:24.152676 8452 raft_consensus.cc:482] T 07965a958d654f208b94dc29b9d2c635 P f85c3f5cab7b4442b6f829cfbadd33b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:18:24.152719 8452 raft_consensus.cc:3037] T 07965a958d654f208b94dc29b9d2c635 P f85c3f5cab7b4442b6f829cfbadd33b5 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:18:24.153703 8452 raft_consensus.cc:504] T 07965a958d654f208b94dc29b9d2c635 P f85c3f5cab7b4442b6f829cfbadd33b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 39557 } } peers { permanent_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39501 } } peers { permanent_uuid: "e94305e8231b4fd9ad7f29754b581d84" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39465 } }
I20260618 22:18:24.154066 8452 leader_election.cc:288] T 07965a958d654f208b94dc29b9d2c635 P f85c3f5cab7b4442b6f829cfbadd33b5 [CANDIDATE]: Term 1 election: Requested vote from peers 2fb18f58d30d4e69998fd6b5bd106bff (127.0.98.67:39501), e94305e8231b4fd9ad7f29754b581d84 (127.0.98.66:39465)
I20260618 22:18:24.154389 8370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07965a958d654f208b94dc29b9d2c635" candidate_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fb18f58d30d4e69998fd6b5bd106bff"
I20260618 22:18:24.154479 8230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07965a958d654f208b94dc29b9d2c635" candidate_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e94305e8231b4fd9ad7f29754b581d84"
I20260618 22:18:24.154525 8370 raft_consensus.cc:3037] T 07965a958d654f208b94dc29b9d2c635 P 2fb18f58d30d4e69998fd6b5bd106bff [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:18:24.154594 8230 raft_consensus.cc:3037] T 07965a958d654f208b94dc29b9d2c635 P e94305e8231b4fd9ad7f29754b581d84 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:18:24.155535 8370 raft_consensus.cc:2454] T 07965a958d654f208b94dc29b9d2c635 P 2fb18f58d30d4e69998fd6b5bd106bff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f85c3f5cab7b4442b6f829cfbadd33b5 in term 1.
I20260618 22:18:24.155746 8230 raft_consensus.cc:2454] T 07965a958d654f208b94dc29b9d2c635 P e94305e8231b4fd9ad7f29754b581d84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f85c3f5cab7b4442b6f829cfbadd33b5 in term 1.
I20260618 22:18:24.155885 8023 leader_election.cc:302] T 07965a958d654f208b94dc29b9d2c635 P f85c3f5cab7b4442b6f829cfbadd33b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fb18f58d30d4e69998fd6b5bd106bff, f85c3f5cab7b4442b6f829cfbadd33b5; no voters:
I20260618 22:18:24.156038 8452 raft_consensus.cc:2781] T 07965a958d654f208b94dc29b9d2c635 P f85c3f5cab7b4442b6f829cfbadd33b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:18:24.156234 8452 raft_consensus.cc:686] T 07965a958d654f208b94dc29b9d2c635 P f85c3f5cab7b4442b6f829cfbadd33b5 [term 1 LEADER]: Becoming Leader. State: Replica: f85c3f5cab7b4442b6f829cfbadd33b5, State: Running, Role: LEADER
I20260618 22:18:24.156574 8452 consensus_queue.cc:237] T 07965a958d654f208b94dc29b9d2c635 P f85c3f5cab7b4442b6f829cfbadd33b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 39557 } } peers { permanent_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39501 } } peers { permanent_uuid: "e94305e8231b4fd9ad7f29754b581d84" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39465 } }
I20260618 22:18:24.159773 7933 catalog_manager.cc:5495] T 07965a958d654f208b94dc29b9d2c635 P f85c3f5cab7b4442b6f829cfbadd33b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to f85c3f5cab7b4442b6f829cfbadd33b5 (127.0.98.65). New cstate: current_term: 1 leader_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 39557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e94305e8231b4fd9ad7f29754b581d84" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39465 } health_report { overall_health: UNKNOWN } } }
I20260618 22:18:24.183398 8439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:24.180179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58358 (local address 127.0.98.65:39557)
0618 22:18:24.180279 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:24.180282 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:24.180294 (+ 12us) server_negotiation.cc:407] Connection header received
0618 22:18:24.180429 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:24.180432 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:24.180481 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:24.180563 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:18:24.180753 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.181228 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.182137 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.182365 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.182446 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:24.182703 (+ 257us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:18:24.183100 (+ 397us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:18:24.183194 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:24.183254 (+ 60us) server_negotiation.cc:299] Negotiation successful
0618 22:18:24.183300 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260618 22:18:24.189354 8370 raft_consensus.cc:1261] T 07965a958d654f208b94dc29b9d2c635 P 2fb18f58d30d4e69998fd6b5bd106bff [term 1 FOLLOWER]: Refusing update from remote peer f85c3f5cab7b4442b6f829cfbadd33b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:18:24.189453 8230 raft_consensus.cc:1261] T 07965a958d654f208b94dc29b9d2c635 P e94305e8231b4fd9ad7f29754b581d84 [term 1 FOLLOWER]: Refusing update from remote peer f85c3f5cab7b4442b6f829cfbadd33b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:18:24.190070 8452 consensus_queue.cc:1035] T 07965a958d654f208b94dc29b9d2c635 P f85c3f5cab7b4442b6f829cfbadd33b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:18:24.190251 8455 consensus_queue.cc:1035] T 07965a958d654f208b94dc29b9d2c635 P f85c3f5cab7b4442b6f829cfbadd33b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e94305e8231b4fd9ad7f29754b581d84" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:18:24.196390 8458 mvcc.cc:205] Tried to move back new op lower bound from 7298339242751754240 to 7298339242628431872. Current Snapshot: MvccSnapshot[applied={T|T < 7298339242751754240}]
I20260618 22:18:24.200148 8459 mvcc.cc:205] Tried to move back new op lower bound from 7298339242751754240 to 7298339242628431872. Current Snapshot: MvccSnapshot[applied={T|T < 7298339242751754240}]
I20260618 22:18:24.204531 7932 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38712:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:18:24.204679 7932 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38712:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:18:24.207141 7932 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ec613fe2b9494be494f0f98c8cfe0d63: Sending DeleteTablet for 3 replicas of tablet 07965a958d654f208b94dc29b9d2c635
I20260618 22:18:24.207836 8070 tablet_service.cc:1514] Processing DeleteTablet for tablet 07965a958d654f208b94dc29b9d2c635 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:18:24 UTC) from {username='oryx', principal='oryx/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:58350
I20260618 22:18:24.208036 8210 tablet_service.cc:1514] Processing DeleteTablet for tablet 07965a958d654f208b94dc29b9d2c635 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:18:24 UTC) from {username='oryx', principal='oryx/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:36894
I20260618 22:18:24.208273 8350 tablet_service.cc:1514] Processing DeleteTablet for tablet 07965a958d654f208b94dc29b9d2c635 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:18:24 UTC) from {username='oryx', principal='oryx/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:45698
I20260618 22:18:24.208372 7932 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38712:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260618 22:18:24.208670 8470 tablet_replica.cc:330] T 07965a958d654f208b94dc29b9d2c635 P 2fb18f58d30d4e69998fd6b5bd106bff: stopping tablet replica
W20260618 22:18:24.208745 7932 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:18:24.208904 8468 tablet_replica.cc:330] T 07965a958d654f208b94dc29b9d2c635 P f85c3f5cab7b4442b6f829cfbadd33b5: stopping tablet replica
I20260618 22:18:24.208979 8470 raft_consensus.cc:2229] T 07965a958d654f208b94dc29b9d2c635 P 2fb18f58d30d4e69998fd6b5bd106bff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:18:24.209118 8468 raft_consensus.cc:2229] T 07965a958d654f208b94dc29b9d2c635 P f85c3f5cab7b4442b6f829cfbadd33b5 [term 1 LEADER]: Raft consensus shutting down.
I20260618 22:18:24.209232 8470 raft_consensus.cc:2258] T 07965a958d654f208b94dc29b9d2c635 P 2fb18f58d30d4e69998fd6b5bd106bff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:18:24.209882 8468 raft_consensus.cc:2258] T 07965a958d654f208b94dc29b9d2c635 P f85c3f5cab7b4442b6f829cfbadd33b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:18:24.210301 8469 tablet_replica.cc:330] T 07965a958d654f208b94dc29b9d2c635 P e94305e8231b4fd9ad7f29754b581d84: stopping tablet replica
I20260618 22:18:24.210593 8470 ts_tablet_manager.cc:1905] T 07965a958d654f208b94dc29b9d2c635 P 2fb18f58d30d4e69998fd6b5bd106bff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:18:24.210604 8469 raft_consensus.cc:2229] T 07965a958d654f208b94dc29b9d2c635 P e94305e8231b4fd9ad7f29754b581d84 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:18:24.210853 8469 raft_consensus.cc:2258] T 07965a958d654f208b94dc29b9d2c635 P e94305e8231b4fd9ad7f29754b581d84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:18:24.211995 8468 ts_tablet_manager.cc:1905] T 07965a958d654f208b94dc29b9d2c635 P f85c3f5cab7b4442b6f829cfbadd33b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:18:24.213678 8470 ts_tablet_manager.cc:1918] T 07965a958d654f208b94dc29b9d2c635 P 2fb18f58d30d4e69998fd6b5bd106bff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:18:24.213794 8470 log.cc:1196] T 07965a958d654f208b94dc29b9d2c635 P 2fb18f58d30d4e69998fd6b5bd106bff: Deleting WAL directory at /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/ts-2/wal/wals/07965a958d654f208b94dc29b9d2c635
I20260618 22:18:24.214129 8470 ts_tablet_manager.cc:1939] T 07965a958d654f208b94dc29b9d2c635 P 2fb18f58d30d4e69998fd6b5bd106bff: Deleting consensus metadata
I20260618 22:18:24.214644 8070 tablet_service.cc:1467] Processing CreateTablet for tablet 4185de86123b419799b2d64becd5f38a (DEFAULT_TABLE table=test-table [id=148d54c0c51e49c5bbf7f567dcebacd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:18:24.214810 8468 ts_tablet_manager.cc:1918] T 07965a958d654f208b94dc29b9d2c635 P f85c3f5cab7b4442b6f829cfbadd33b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:18:24.214887 8468 log.cc:1196] T 07965a958d654f208b94dc29b9d2c635 P f85c3f5cab7b4442b6f829cfbadd33b5: Deleting WAL directory at /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/ts-0/wal/wals/07965a958d654f208b94dc29b9d2c635
I20260618 22:18:24.214918 8070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4185de86123b419799b2d64becd5f38a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:18:24.215096 8469 ts_tablet_manager.cc:1905] T 07965a958d654f208b94dc29b9d2c635 P e94305e8231b4fd9ad7f29754b581d84: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:18:24.215186 8468 ts_tablet_manager.cc:1939] T 07965a958d654f208b94dc29b9d2c635 P f85c3f5cab7b4442b6f829cfbadd33b5: Deleting consensus metadata
I20260618 22:18:24.216820 8210 tablet_service.cc:1467] Processing CreateTablet for tablet 4185de86123b419799b2d64becd5f38a (DEFAULT_TABLE table=test-table [id=148d54c0c51e49c5bbf7f567dcebacd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:18:24.217048 8210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4185de86123b419799b2d64becd5f38a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:18:24.217109 8445 tablet_bootstrap.cc:492] T 4185de86123b419799b2d64becd5f38a P f85c3f5cab7b4442b6f829cfbadd33b5: Bootstrap starting.
I20260618 22:18:24.217588 8469 ts_tablet_manager.cc:1918] T 07965a958d654f208b94dc29b9d2c635 P e94305e8231b4fd9ad7f29754b581d84: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:18:24.217661 8469 log.cc:1196] T 07965a958d654f208b94dc29b9d2c635 P e94305e8231b4fd9ad7f29754b581d84: Deleting WAL directory at /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/ts-1/wal/wals/07965a958d654f208b94dc29b9d2c635
I20260618 22:18:24.218034 8445 tablet_bootstrap.cc:654] T 4185de86123b419799b2d64becd5f38a P f85c3f5cab7b4442b6f829cfbadd33b5: Neither blocks nor log segments found. Creating new log.
I20260618 22:18:24.218959 8469 ts_tablet_manager.cc:1939] T 07965a958d654f208b94dc29b9d2c635 P e94305e8231b4fd9ad7f29754b581d84: Deleting consensus metadata
I20260618 22:18:24.219025 7919 catalog_manager.cc:4841] TS 2fb18f58d30d4e69998fd6b5bd106bff (127.0.98.67:39501): tablet 07965a958d654f208b94dc29b9d2c635 (table test-table [id=ad25bd072a4a471892d47750d1cf9e1b]) successfully deleted
I20260618 22:18:24.219252 8446 tablet_bootstrap.cc:492] T 4185de86123b419799b2d64becd5f38a P e94305e8231b4fd9ad7f29754b581d84: Bootstrap starting.
I20260618 22:18:24.219646 7918 catalog_manager.cc:4841] TS f85c3f5cab7b4442b6f829cfbadd33b5 (127.0.98.65:39557): tablet 07965a958d654f208b94dc29b9d2c635 (table test-table [id=ad25bd072a4a471892d47750d1cf9e1b]) successfully deleted
I20260618 22:18:24.216511 8350 tablet_service.cc:1467] Processing CreateTablet for tablet 4185de86123b419799b2d64becd5f38a (DEFAULT_TABLE table=test-table [id=148d54c0c51e49c5bbf7f567dcebacd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:18:24.219996 7919 catalog_manager.cc:4841] TS e94305e8231b4fd9ad7f29754b581d84 (127.0.98.66:39465): tablet 07965a958d654f208b94dc29b9d2c635 (table test-table [id=ad25bd072a4a471892d47750d1cf9e1b]) successfully deleted
I20260618 22:18:24.220039 8350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4185de86123b419799b2d64becd5f38a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:18:24.220274 8446 tablet_bootstrap.cc:654] T 4185de86123b419799b2d64becd5f38a P e94305e8231b4fd9ad7f29754b581d84: Neither blocks nor log segments found. Creating new log.
I20260618 22:18:24.221419 8446 tablet_bootstrap.cc:492] T 4185de86123b419799b2d64becd5f38a P e94305e8231b4fd9ad7f29754b581d84: No bootstrap required, opened a new log
I20260618 22:18:24.221500 8446 ts_tablet_manager.cc:1397] T 4185de86123b419799b2d64becd5f38a P e94305e8231b4fd9ad7f29754b581d84: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260618 22:18:24.222003 8446 raft_consensus.cc:348] T 4185de86123b419799b2d64becd5f38a P e94305e8231b4fd9ad7f29754b581d84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 39557 } } peers { permanent_uuid: "e94305e8231b4fd9ad7f29754b581d84" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39465 } } peers { permanent_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39501 } }
I20260618 22:18:24.222116 8446 raft_consensus.cc:374] T 4185de86123b419799b2d64becd5f38a P e94305e8231b4fd9ad7f29754b581d84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:18:24.222158 8446 raft_consensus.cc:729] T 4185de86123b419799b2d64becd5f38a P e94305e8231b4fd9ad7f29754b581d84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e94305e8231b4fd9ad7f29754b581d84, State: Initialized, Role: FOLLOWER
I20260618 22:18:24.222178 8444 tablet_bootstrap.cc:492] T 4185de86123b419799b2d64becd5f38a P 2fb18f58d30d4e69998fd6b5bd106bff: Bootstrap starting.
I20260618 22:18:24.222292 8446 consensus_queue.cc:260] T 4185de86123b419799b2d64becd5f38a P e94305e8231b4fd9ad7f29754b581d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 39557 } } peers { permanent_uuid: "e94305e8231b4fd9ad7f29754b581d84" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39465 } } peers { permanent_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39501 } }
I20260618 22:18:24.222576 8446 ts_tablet_manager.cc:1428] T 4185de86123b419799b2d64becd5f38a P e94305e8231b4fd9ad7f29754b581d84: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260618 22:18:24.223088 8444 tablet_bootstrap.cc:654] T 4185de86123b419799b2d64becd5f38a P 2fb18f58d30d4e69998fd6b5bd106bff: Neither blocks nor log segments found. Creating new log.
I20260618 22:18:24.224294 8445 tablet_bootstrap.cc:492] T 4185de86123b419799b2d64becd5f38a P f85c3f5cab7b4442b6f829cfbadd33b5: No bootstrap required, opened a new log
I20260618 22:18:24.224371 8445 ts_tablet_manager.cc:1397] T 4185de86123b419799b2d64becd5f38a P f85c3f5cab7b4442b6f829cfbadd33b5: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20260618 22:18:24.224717 8444 tablet_bootstrap.cc:492] T 4185de86123b419799b2d64becd5f38a P 2fb18f58d30d4e69998fd6b5bd106bff: No bootstrap required, opened a new log
I20260618 22:18:24.224774 8444 ts_tablet_manager.cc:1397] T 4185de86123b419799b2d64becd5f38a P 2fb18f58d30d4e69998fd6b5bd106bff: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260618 22:18:24.224773 8445 raft_consensus.cc:348] T 4185de86123b419799b2d64becd5f38a P f85c3f5cab7b4442b6f829cfbadd33b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 39557 } } peers { permanent_uuid: "e94305e8231b4fd9ad7f29754b581d84" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39465 } } peers { permanent_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39501 } }
I20260618 22:18:24.224865 8445 raft_consensus.cc:374] T 4185de86123b419799b2d64becd5f38a P f85c3f5cab7b4442b6f829cfbadd33b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:18:24.224895 8445 raft_consensus.cc:729] T 4185de86123b419799b2d64becd5f38a P f85c3f5cab7b4442b6f829cfbadd33b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f85c3f5cab7b4442b6f829cfbadd33b5, State: Initialized, Role: FOLLOWER
I20260618 22:18:24.225004 8445 consensus_queue.cc:260] T 4185de86123b419799b2d64becd5f38a P f85c3f5cab7b4442b6f829cfbadd33b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 39557 } } peers { permanent_uuid: "e94305e8231b4fd9ad7f29754b581d84" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39465 } } peers { permanent_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39501 } }
I20260618 22:18:24.225224 8445 ts_tablet_manager.cc:1428] T 4185de86123b419799b2d64becd5f38a P f85c3f5cab7b4442b6f829cfbadd33b5: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260618 22:18:24.225173 8444 raft_consensus.cc:348] T 4185de86123b419799b2d64becd5f38a P 2fb18f58d30d4e69998fd6b5bd106bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 39557 } } peers { permanent_uuid: "e94305e8231b4fd9ad7f29754b581d84" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39465 } } peers { permanent_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39501 } }
I20260618 22:18:24.225279 8444 raft_consensus.cc:374] T 4185de86123b419799b2d64becd5f38a P 2fb18f58d30d4e69998fd6b5bd106bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:18:24.225311 8444 raft_consensus.cc:729] T 4185de86123b419799b2d64becd5f38a P 2fb18f58d30d4e69998fd6b5bd106bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fb18f58d30d4e69998fd6b5bd106bff, State: Initialized, Role: FOLLOWER
I20260618 22:18:24.225431 8444 consensus_queue.cc:260] T 4185de86123b419799b2d64becd5f38a P 2fb18f58d30d4e69998fd6b5bd106bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 39557 } } peers { permanent_uuid: "e94305e8231b4fd9ad7f29754b581d84" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39465 } } peers { permanent_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39501 } }
I20260618 22:18:24.225724 8444 ts_tablet_manager.cc:1428] T 4185de86123b419799b2d64becd5f38a P 2fb18f58d30d4e69998fd6b5bd106bff: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
W20260618 22:18:24.270349 8417 tablet.cc:2369] T 4185de86123b419799b2d64becd5f38a P 2fb18f58d30d4e69998fd6b5bd106bff: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260618 22:18:24.312597 8277 tablet.cc:2369] T 4185de86123b419799b2d64becd5f38a P e94305e8231b4fd9ad7f29754b581d84: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:18:24.336225 8450 raft_consensus.cc:482] T 4185de86123b419799b2d64becd5f38a P 2fb18f58d30d4e69998fd6b5bd106bff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:18:24.336388 8450 raft_consensus.cc:504] T 4185de86123b419799b2d64becd5f38a P 2fb18f58d30d4e69998fd6b5bd106bff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 39557 } } peers { permanent_uuid: "e94305e8231b4fd9ad7f29754b581d84" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39465 } } peers { permanent_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39501 } }
W20260618 22:18:24.336966 8137 tablet.cc:2369] T 4185de86123b419799b2d64becd5f38a P f85c3f5cab7b4442b6f829cfbadd33b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:18:24.337519 8450 leader_election.cc:288] T 4185de86123b419799b2d64becd5f38a P 2fb18f58d30d4e69998fd6b5bd106bff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f85c3f5cab7b4442b6f829cfbadd33b5 (127.0.98.65:39557), e94305e8231b4fd9ad7f29754b581d84 (127.0.98.66:39465)
I20260618 22:18:24.342002 8443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:24.338257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:58899 (local address 127.0.98.66:39465)
0618 22:18:24.338378 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:24.338383 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:24.338398 (+ 15us) server_negotiation.cc:407] Connection header received
0618 22:18:24.338450 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:24.338453 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:24.338502 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:24.338598 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:18:24.339027 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.339629 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.340721 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.341311 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.341427 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:24.341524 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:24.341766 (+ 242us) server_negotiation.cc:299] Negotiation successful
0618 22:18:24.341857 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260618 22:18:24.342154 8439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:24.337910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:39357 (local address 127.0.98.65:39557)
0618 22:18:24.338025 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:24.338032 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:24.338049 (+ 17us) server_negotiation.cc:407] Connection header received
0618 22:18:24.338104 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:24.338107 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:24.338148 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:24.338215 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:18:24.338652 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.339312 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.340874 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.341435 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.341523 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:24.341590 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:24.341973 (+ 383us) server_negotiation.cc:299] Negotiation successful
0618 22:18:24.342049 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260618 22:18:24.341987 8474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:24.337875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:39465 (local address 127.0.98.67:58899)
0618 22:18:24.338224 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:24.338236 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:24.338313 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:24.338724 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:24.338727 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:24.338746 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:18:24.338889 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:24.338896 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.339734 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.339739 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:24.340598 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:24.340607 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.341473 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.341477 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:24.341576 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:24.341604 (+ 28us) client_negotiation.cc:769] Sending connection context
0618 22:18:24.341724 (+ 120us) client_negotiation.cc:240] Negotiation successful
0618 22:18:24.341797 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"mutex_wait_us":14,"thread_start_us":83,"threads_started":1}
I20260618 22:18:24.342515 8309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:24.337703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:39557 (local address 127.0.98.67:39357)
0618 22:18:24.337858 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:24.337876 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:24.337996 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:24.338334 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:24.338336 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:24.338353 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:18:24.338519 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:24.338529 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.339446 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.339457 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:24.340743 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:24.340754 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.341594 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.341598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:24.341689 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:24.341741 (+ 52us) client_negotiation.cc:769] Sending connection context
0618 22:18:24.341810 (+ 69us) client_negotiation.cc:240] Negotiation successful
0618 22:18:24.341874 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76,"mutex_wait_us":1}
I20260618 22:18:24.343025 8230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4185de86123b419799b2d64becd5f38a" candidate_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e94305e8231b4fd9ad7f29754b581d84" is_pre_election: true
I20260618 22:18:24.343107 8090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4185de86123b419799b2d64becd5f38a" candidate_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" is_pre_election: true
I20260618 22:18:24.343200 8230 raft_consensus.cc:2454] T 4185de86123b419799b2d64becd5f38a P e94305e8231b4fd9ad7f29754b581d84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fb18f58d30d4e69998fd6b5bd106bff in term 0.
I20260618 22:18:24.343334 8090 raft_consensus.cc:2454] T 4185de86123b419799b2d64becd5f38a P f85c3f5cab7b4442b6f829cfbadd33b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fb18f58d30d4e69998fd6b5bd106bff in term 0.
I20260618 22:18:24.343549 8303 leader_election.cc:302] T 4185de86123b419799b2d64becd5f38a P 2fb18f58d30d4e69998fd6b5bd106bff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fb18f58d30d4e69998fd6b5bd106bff, e94305e8231b4fd9ad7f29754b581d84; no voters:
I20260618 22:18:24.343927 8450 raft_consensus.cc:2781] T 4185de86123b419799b2d64becd5f38a P 2fb18f58d30d4e69998fd6b5bd106bff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:18:24.344002 8450 raft_consensus.cc:482] T 4185de86123b419799b2d64becd5f38a P 2fb18f58d30d4e69998fd6b5bd106bff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:18:24.344055 8450 raft_consensus.cc:3037] T 4185de86123b419799b2d64becd5f38a P 2fb18f58d30d4e69998fd6b5bd106bff [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:18:24.345000 8450 raft_consensus.cc:504] T 4185de86123b419799b2d64becd5f38a P 2fb18f58d30d4e69998fd6b5bd106bff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 39557 } } peers { permanent_uuid: "e94305e8231b4fd9ad7f29754b581d84" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39465 } } peers { permanent_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39501 } }
I20260618 22:18:24.345374 8450 leader_election.cc:288] T 4185de86123b419799b2d64becd5f38a P 2fb18f58d30d4e69998fd6b5bd106bff [CANDIDATE]: Term 1 election: Requested vote from peers f85c3f5cab7b4442b6f829cfbadd33b5 (127.0.98.65:39557), e94305e8231b4fd9ad7f29754b581d84 (127.0.98.66:39465)
I20260618 22:18:24.345608 8090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4185de86123b419799b2d64becd5f38a" candidate_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5"
I20260618 22:18:24.345710 8090 raft_consensus.cc:3037] T 4185de86123b419799b2d64becd5f38a P f85c3f5cab7b4442b6f829cfbadd33b5 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:18:24.345731 8230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4185de86123b419799b2d64becd5f38a" candidate_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e94305e8231b4fd9ad7f29754b581d84"
I20260618 22:18:24.345834 8230 raft_consensus.cc:3037] T 4185de86123b419799b2d64becd5f38a P e94305e8231b4fd9ad7f29754b581d84 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:18:24.346886 8090 raft_consensus.cc:2454] T 4185de86123b419799b2d64becd5f38a P f85c3f5cab7b4442b6f829cfbadd33b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fb18f58d30d4e69998fd6b5bd106bff in term 1.
I20260618 22:18:24.346903 8230 raft_consensus.cc:2454] T 4185de86123b419799b2d64becd5f38a P e94305e8231b4fd9ad7f29754b581d84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fb18f58d30d4e69998fd6b5bd106bff in term 1.
I20260618 22:18:24.347175 8302 leader_election.cc:302] T 4185de86123b419799b2d64becd5f38a P 2fb18f58d30d4e69998fd6b5bd106bff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fb18f58d30d4e69998fd6b5bd106bff, f85c3f5cab7b4442b6f829cfbadd33b5; no voters:
I20260618 22:18:24.347348 8450 raft_consensus.cc:2781] T 4185de86123b419799b2d64becd5f38a P 2fb18f58d30d4e69998fd6b5bd106bff [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:18:24.347576 8450 raft_consensus.cc:686] T 4185de86123b419799b2d64becd5f38a P 2fb18f58d30d4e69998fd6b5bd106bff [term 1 LEADER]: Becoming Leader. State: Replica: 2fb18f58d30d4e69998fd6b5bd106bff, State: Running, Role: LEADER
I20260618 22:18:24.347813 8450 consensus_queue.cc:237] T 4185de86123b419799b2d64becd5f38a P 2fb18f58d30d4e69998fd6b5bd106bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 39557 } } peers { permanent_uuid: "e94305e8231b4fd9ad7f29754b581d84" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39465 } } peers { permanent_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39501 } }
I20260618 22:18:24.350071 7931 catalog_manager.cc:5495] T 4185de86123b419799b2d64becd5f38a P 2fb18f58d30d4e69998fd6b5bd106bff reported cstate change: term changed from 0 to 1, leader changed from <none> to 2fb18f58d30d4e69998fd6b5bd106bff (127.0.98.67). New cstate: current_term: 1 leader_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 39557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e94305e8231b4fd9ad7f29754b581d84" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39501 } health_report { overall_health: HEALTHY } } }
Jun 18 22:18:24 dist-test-slave-g6r4 krb5kdc[7888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821103, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.126@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM
I20260618 22:18:24.415634 8482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:24.407625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37239 (local address 127.0.0.1:38716)
0618 22:18:24.407921 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:24.407936 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:24.408051 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:24.408293 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:24.408296 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:24.408683 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:24.408773 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:24.408778 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.409515 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.409517 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:24.410086 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:24.410094 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.410526 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.410529 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:24.410587 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:24.411142 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:24.411153 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:24.412497 (+ 1344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:18:24.414116 (+ 1619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:24.414124 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:24.414129 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:24.414364 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:24.414695 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:24.414697 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:24.414698 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:24.414775 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:24.415127 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:18:24.415132 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:18:24.415227 (+ 95us) client_negotiation.cc:769] Sending connection context
0618 22:18:24.415314 (+ 87us) client_negotiation.cc:240] Negotiation successful
0618 22:18:24.415470 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":132,"threads_started":1}
I20260618 22:18:24.415843 7982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:24.407665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38716 (local address 127.0.98.126:37239)
0618 22:18:24.407785 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:24.407789 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:24.408021 (+ 232us) server_negotiation.cc:407] Connection header received
0618 22:18:24.408141 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:24.408143 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:24.408180 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:24.408258 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:24.408889 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.409419 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.410213 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.410432 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.410512 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:24.412636 (+ 2124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:24.412639 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:24.412641 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:24.412661 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:24.414024 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:24.414475 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:24.414478 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:24.414479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:24.414565 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:24.414857 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:24.414860 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:24.414861 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:24.415035 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:24.415127 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:24.415414 (+ 287us) server_negotiation.cc:299] Negotiation successful
0618 22:18:24.415523 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260618 22:18:24.419289 7931 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:38716:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260618 22:18:24.420038 7931 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:18:24.424664 8070 tablet_service.cc:1467] Processing CreateTablet for tablet 764f2584fbfc45659d9fcdae3b0ef858 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=48f0a861787b42edad2d9282bceccaf5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260618 22:18:24.424695 8210 tablet_service.cc:1467] Processing CreateTablet for tablet 764f2584fbfc45659d9fcdae3b0ef858 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=48f0a861787b42edad2d9282bceccaf5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260618 22:18:24.425050 8210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 764f2584fbfc45659d9fcdae3b0ef858. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:18:24.425050 8070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 764f2584fbfc45659d9fcdae3b0ef858. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:18:24.424919 8350 tablet_service.cc:1467] Processing CreateTablet for tablet 764f2584fbfc45659d9fcdae3b0ef858 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=48f0a861787b42edad2d9282bceccaf5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260618 22:18:24.425319 8350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 764f2584fbfc45659d9fcdae3b0ef858. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:18:24.427759 8445 tablet_bootstrap.cc:492] T 764f2584fbfc45659d9fcdae3b0ef858 P f85c3f5cab7b4442b6f829cfbadd33b5: Bootstrap starting.
I20260618 22:18:24.427793 8444 tablet_bootstrap.cc:492] T 764f2584fbfc45659d9fcdae3b0ef858 P 2fb18f58d30d4e69998fd6b5bd106bff: Bootstrap starting.
I20260618 22:18:24.428087 8446 tablet_bootstrap.cc:492] T 764f2584fbfc45659d9fcdae3b0ef858 P e94305e8231b4fd9ad7f29754b581d84: Bootstrap starting.
I20260618 22:18:24.428601 8445 tablet_bootstrap.cc:654] T 764f2584fbfc45659d9fcdae3b0ef858 P f85c3f5cab7b4442b6f829cfbadd33b5: Neither blocks nor log segments found. Creating new log.
I20260618 22:18:24.428787 8444 tablet_bootstrap.cc:654] T 764f2584fbfc45659d9fcdae3b0ef858 P 2fb18f58d30d4e69998fd6b5bd106bff: Neither blocks nor log segments found. Creating new log.
I20260618 22:18:24.429001 8446 tablet_bootstrap.cc:654] T 764f2584fbfc45659d9fcdae3b0ef858 P e94305e8231b4fd9ad7f29754b581d84: Neither blocks nor log segments found. Creating new log.
I20260618 22:18:24.429862 8445 tablet_bootstrap.cc:492] T 764f2584fbfc45659d9fcdae3b0ef858 P f85c3f5cab7b4442b6f829cfbadd33b5: No bootstrap required, opened a new log
I20260618 22:18:24.429992 8445 ts_tablet_manager.cc:1397] T 764f2584fbfc45659d9fcdae3b0ef858 P f85c3f5cab7b4442b6f829cfbadd33b5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260618 22:18:24.430369 8445 raft_consensus.cc:348] T 764f2584fbfc45659d9fcdae3b0ef858 P f85c3f5cab7b4442b6f829cfbadd33b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 39557 } } peers { permanent_uuid: "e94305e8231b4fd9ad7f29754b581d84" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39465 } } peers { permanent_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39501 } }
I20260618 22:18:24.430452 8445 raft_consensus.cc:374] T 764f2584fbfc45659d9fcdae3b0ef858 P f85c3f5cab7b4442b6f829cfbadd33b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:18:24.430473 8445 raft_consensus.cc:729] T 764f2584fbfc45659d9fcdae3b0ef858 P f85c3f5cab7b4442b6f829cfbadd33b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f85c3f5cab7b4442b6f829cfbadd33b5, State: Initialized, Role: FOLLOWER
I20260618 22:18:24.430563 8444 tablet_bootstrap.cc:492] T 764f2584fbfc45659d9fcdae3b0ef858 P 2fb18f58d30d4e69998fd6b5bd106bff: No bootstrap required, opened a new log
I20260618 22:18:24.430563 8446 tablet_bootstrap.cc:492] T 764f2584fbfc45659d9fcdae3b0ef858 P e94305e8231b4fd9ad7f29754b581d84: No bootstrap required, opened a new log
I20260618 22:18:24.430641 8444 ts_tablet_manager.cc:1397] T 764f2584fbfc45659d9fcdae3b0ef858 P 2fb18f58d30d4e69998fd6b5bd106bff: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260618 22:18:24.430641 8446 ts_tablet_manager.cc:1397] T 764f2584fbfc45659d9fcdae3b0ef858 P e94305e8231b4fd9ad7f29754b581d84: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260618 22:18:24.430600 8445 consensus_queue.cc:260] T 764f2584fbfc45659d9fcdae3b0ef858 P f85c3f5cab7b4442b6f829cfbadd33b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 39557 } } peers { permanent_uuid: "e94305e8231b4fd9ad7f29754b581d84" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39465 } } peers { permanent_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39501 } }
I20260618 22:18:24.430909 8445 ts_tablet_manager.cc:1428] T 764f2584fbfc45659d9fcdae3b0ef858 P f85c3f5cab7b4442b6f829cfbadd33b5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260618 22:18:24.430873 8463 tablet_replica.cc:439] T 764f2584fbfc45659d9fcdae3b0ef858 P f85c3f5cab7b4442b6f829cfbadd33b5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 39557 } } peers { permanent_uuid: "e94305e8231b4fd9ad7f29754b581d84" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39465 } } peers { permanent_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39501 } } }
I20260618 22:18:24.431001 8463 tablet_replica.cc:442] T 764f2584fbfc45659d9fcdae3b0ef858 P f85c3f5cab7b4442b6f829cfbadd33b5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 22:18:24.431116 8446 raft_consensus.cc:348] T 764f2584fbfc45659d9fcdae3b0ef858 P e94305e8231b4fd9ad7f29754b581d84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 39557 } } peers { permanent_uuid: "e94305e8231b4fd9ad7f29754b581d84" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39465 } } peers { permanent_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39501 } }
I20260618 22:18:24.431116 8444 raft_consensus.cc:348] T 764f2584fbfc45659d9fcdae3b0ef858 P 2fb18f58d30d4e69998fd6b5bd106bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 39557 } } peers { permanent_uuid: "e94305e8231b4fd9ad7f29754b581d84" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39465 } } peers { permanent_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39501 } }
I20260618 22:18:24.431242 8446 raft_consensus.cc:374] T 764f2584fbfc45659d9fcdae3b0ef858 P e94305e8231b4fd9ad7f29754b581d84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:18:24.431255 8444 raft_consensus.cc:374] T 764f2584fbfc45659d9fcdae3b0ef858 P 2fb18f58d30d4e69998fd6b5bd106bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:18:24.431288 8446 raft_consensus.cc:729] T 764f2584fbfc45659d9fcdae3b0ef858 P e94305e8231b4fd9ad7f29754b581d84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e94305e8231b4fd9ad7f29754b581d84, State: Initialized, Role: FOLLOWER
I20260618 22:18:24.431294 8444 raft_consensus.cc:729] T 764f2584fbfc45659d9fcdae3b0ef858 P 2fb18f58d30d4e69998fd6b5bd106bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fb18f58d30d4e69998fd6b5bd106bff, State: Initialized, Role: FOLLOWER
I20260618 22:18:24.431432 8446 consensus_queue.cc:260] T 764f2584fbfc45659d9fcdae3b0ef858 P e94305e8231b4fd9ad7f29754b581d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 39557 } } peers { permanent_uuid: "e94305e8231b4fd9ad7f29754b581d84" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39465 } } peers { permanent_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39501 } }
I20260618 22:18:24.431514 8444 consensus_queue.cc:260] T 764f2584fbfc45659d9fcdae3b0ef858 P 2fb18f58d30d4e69998fd6b5bd106bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 39557 } } peers { permanent_uuid: "e94305e8231b4fd9ad7f29754b581d84" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39465 } } peers { permanent_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39501 } }
I20260618 22:18:24.431743 8446 ts_tablet_manager.cc:1428] T 764f2584fbfc45659d9fcdae3b0ef858 P e94305e8231b4fd9ad7f29754b581d84: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260618 22:18:24.431692 8451 tablet_replica.cc:439] T 764f2584fbfc45659d9fcdae3b0ef858 P e94305e8231b4fd9ad7f29754b581d84: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 39557 } } peers { permanent_uuid: "e94305e8231b4fd9ad7f29754b581d84" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39465 } } peers { permanent_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39501 } } }
I20260618 22:18:24.431766 8450 tablet_replica.cc:439] T 764f2584fbfc45659d9fcdae3b0ef858 P 2fb18f58d30d4e69998fd6b5bd106bff: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 39557 } } peers { permanent_uuid: "e94305e8231b4fd9ad7f29754b581d84" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39465 } } peers { permanent_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39501 } } }
I20260618 22:18:24.431847 8451 tablet_replica.cc:442] T 764f2584fbfc45659d9fcdae3b0ef858 P e94305e8231b4fd9ad7f29754b581d84: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 22:18:24.431878 8450 tablet_replica.cc:442] T 764f2584fbfc45659d9fcdae3b0ef858 P 2fb18f58d30d4e69998fd6b5bd106bff: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 22:18:24.432567 8444 ts_tablet_manager.cc:1428] T 764f2584fbfc45659d9fcdae3b0ef858 P 2fb18f58d30d4e69998fd6b5bd106bff: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260618 22:18:24.465862 8450 raft_consensus.cc:482] T 764f2584fbfc45659d9fcdae3b0ef858 P 2fb18f58d30d4e69998fd6b5bd106bff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:18:24.466046 8450 raft_consensus.cc:504] T 764f2584fbfc45659d9fcdae3b0ef858 P 2fb18f58d30d4e69998fd6b5bd106bff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 39557 } } peers { permanent_uuid: "e94305e8231b4fd9ad7f29754b581d84" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39465 } } peers { permanent_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39501 } }
I20260618 22:18:24.466495 8450 leader_election.cc:288] T 764f2584fbfc45659d9fcdae3b0ef858 P 2fb18f58d30d4e69998fd6b5bd106bff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f85c3f5cab7b4442b6f829cfbadd33b5 (127.0.98.65:39557), e94305e8231b4fd9ad7f29754b581d84 (127.0.98.66:39465)
I20260618 22:18:24.466933 8230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764f2584fbfc45659d9fcdae3b0ef858" candidate_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e94305e8231b4fd9ad7f29754b581d84" is_pre_election: true
I20260618 22:18:24.466952 8090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764f2584fbfc45659d9fcdae3b0ef858" candidate_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" is_pre_election: true
I20260618 22:18:24.467099 8090 raft_consensus.cc:2454] T 764f2584fbfc45659d9fcdae3b0ef858 P f85c3f5cab7b4442b6f829cfbadd33b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fb18f58d30d4e69998fd6b5bd106bff in term 0.
I20260618 22:18:24.467111 8230 raft_consensus.cc:2454] T 764f2584fbfc45659d9fcdae3b0ef858 P e94305e8231b4fd9ad7f29754b581d84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fb18f58d30d4e69998fd6b5bd106bff in term 0.
I20260618 22:18:24.467415 8302 leader_election.cc:302] T 764f2584fbfc45659d9fcdae3b0ef858 P 2fb18f58d30d4e69998fd6b5bd106bff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fb18f58d30d4e69998fd6b5bd106bff, f85c3f5cab7b4442b6f829cfbadd33b5; no voters:
I20260618 22:18:24.467602 8450 raft_consensus.cc:2781] T 764f2584fbfc45659d9fcdae3b0ef858 P 2fb18f58d30d4e69998fd6b5bd106bff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:18:24.467697 8450 raft_consensus.cc:482] T 764f2584fbfc45659d9fcdae3b0ef858 P 2fb18f58d30d4e69998fd6b5bd106bff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:18:24.467770 8450 raft_consensus.cc:3037] T 764f2584fbfc45659d9fcdae3b0ef858 P 2fb18f58d30d4e69998fd6b5bd106bff [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:18:24.468946 8450 raft_consensus.cc:504] T 764f2584fbfc45659d9fcdae3b0ef858 P 2fb18f58d30d4e69998fd6b5bd106bff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 39557 } } peers { permanent_uuid: "e94305e8231b4fd9ad7f29754b581d84" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39465 } } peers { permanent_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39501 } }
I20260618 22:18:24.469317 8450 leader_election.cc:288] T 764f2584fbfc45659d9fcdae3b0ef858 P 2fb18f58d30d4e69998fd6b5bd106bff [CANDIDATE]: Term 1 election: Requested vote from peers f85c3f5cab7b4442b6f829cfbadd33b5 (127.0.98.65:39557), e94305e8231b4fd9ad7f29754b581d84 (127.0.98.66:39465)
I20260618 22:18:24.469635 8090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764f2584fbfc45659d9fcdae3b0ef858" candidate_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5"
I20260618 22:18:24.469681 8230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "764f2584fbfc45659d9fcdae3b0ef858" candidate_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e94305e8231b4fd9ad7f29754b581d84"
I20260618 22:18:24.469735 8090 raft_consensus.cc:3037] T 764f2584fbfc45659d9fcdae3b0ef858 P f85c3f5cab7b4442b6f829cfbadd33b5 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:18:24.469834 8230 raft_consensus.cc:3037] T 764f2584fbfc45659d9fcdae3b0ef858 P e94305e8231b4fd9ad7f29754b581d84 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:18:24.470667 8090 raft_consensus.cc:2454] T 764f2584fbfc45659d9fcdae3b0ef858 P f85c3f5cab7b4442b6f829cfbadd33b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fb18f58d30d4e69998fd6b5bd106bff in term 1.
I20260618 22:18:24.470774 8230 raft_consensus.cc:2454] T 764f2584fbfc45659d9fcdae3b0ef858 P e94305e8231b4fd9ad7f29754b581d84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fb18f58d30d4e69998fd6b5bd106bff in term 1.
I20260618 22:18:24.470996 8302 leader_election.cc:302] T 764f2584fbfc45659d9fcdae3b0ef858 P 2fb18f58d30d4e69998fd6b5bd106bff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fb18f58d30d4e69998fd6b5bd106bff, f85c3f5cab7b4442b6f829cfbadd33b5; no voters:
I20260618 22:18:24.471180 8450 raft_consensus.cc:2781] T 764f2584fbfc45659d9fcdae3b0ef858 P 2fb18f58d30d4e69998fd6b5bd106bff [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:18:24.471299 8450 raft_consensus.cc:686] T 764f2584fbfc45659d9fcdae3b0ef858 P 2fb18f58d30d4e69998fd6b5bd106bff [term 1 LEADER]: Becoming Leader. State: Replica: 2fb18f58d30d4e69998fd6b5bd106bff, State: Running, Role: LEADER
I20260618 22:18:24.471444 8450 consensus_queue.cc:237] T 764f2584fbfc45659d9fcdae3b0ef858 P 2fb18f58d30d4e69998fd6b5bd106bff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 39557 } } peers { permanent_uuid: "e94305e8231b4fd9ad7f29754b581d84" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39465 } } peers { permanent_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39501 } }
I20260618 22:18:24.472133 8475 tablet_replica.cc:439] T 764f2584fbfc45659d9fcdae3b0ef858 P 2fb18f58d30d4e69998fd6b5bd106bff: TxnStatusTablet state changed. Reason: New leader 2fb18f58d30d4e69998fd6b5bd106bff. Latest consensus state: current_term: 1 leader_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 39557 } } peers { permanent_uuid: "e94305e8231b4fd9ad7f29754b581d84" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39465 } } peers { permanent_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39501 } } }
I20260618 22:18:24.472239 8475 tablet_replica.cc:442] T 764f2584fbfc45659d9fcdae3b0ef858 P 2fb18f58d30d4e69998fd6b5bd106bff: This TxnStatusTablet replica's current role is: LEADER
I20260618 22:18:24.472682 8487 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260618 22:18:24.472851 8487 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260618 22:18:24.473248 7933 catalog_manager.cc:5495] T 764f2584fbfc45659d9fcdae3b0ef858 P 2fb18f58d30d4e69998fd6b5bd106bff reported cstate change: term changed from 0 to 1, leader changed from <none> to 2fb18f58d30d4e69998fd6b5bd106bff (127.0.98.67). New cstate: current_term: 1 leader_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 39557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e94305e8231b4fd9ad7f29754b581d84" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39501 } health_report { overall_health: HEALTHY } } }
I20260618 22:18:24.488299 8482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:24.484157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:39501 (local address 127.0.0.1:45710)
0618 22:18:24.484278 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:24.484294 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:24.484388 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:24.484721 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:24.484730 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:24.484739 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0618 22:18:24.484861 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:24.484866 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.485939 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.485943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:24.486611 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:24.486620 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.487202 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.487209 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:24.487301 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:24.487341 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0618 22:18:24.488031 (+ 690us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0618 22:18:24.488049 (+ 18us) client_negotiation.cc:769] Sending connection context
0618 22:18:24.488105 (+ 56us) client_negotiation.cc:240] Negotiation successful
0618 22:18:24.488172 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260618 22:18:24.488320 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:24.484252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45710 (local address 127.0.98.67:39501)
0618 22:18:24.484375 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:24.484380 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:24.484394 (+ 14us) server_negotiation.cc:407] Connection header received
0618 22:18:24.484515 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:24.484518 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:24.484580 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:24.484685 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:18:24.484971 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.485789 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.486730 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.487069 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.487194 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:24.487466 (+ 272us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:18:24.487936 (+ 470us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:18:24.488047 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:24.488130 (+ 83us) server_negotiation.cc:299] Negotiation successful
0618 22:18:24.488184 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260618 22:18:24.491082 8230 raft_consensus.cc:1261] T 764f2584fbfc45659d9fcdae3b0ef858 P e94305e8231b4fd9ad7f29754b581d84 [term 1 FOLLOWER]: Refusing update from remote peer 2fb18f58d30d4e69998fd6b5bd106bff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:18:24.491114 8090 raft_consensus.cc:1261] T 764f2584fbfc45659d9fcdae3b0ef858 P f85c3f5cab7b4442b6f829cfbadd33b5 [term 1 FOLLOWER]: Refusing update from remote peer 2fb18f58d30d4e69998fd6b5bd106bff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:18:24.491506 8450 consensus_queue.cc:1035] T 764f2584fbfc45659d9fcdae3b0ef858 P 2fb18f58d30d4e69998fd6b5bd106bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "e94305e8231b4fd9ad7f29754b581d84" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:18:24.491721 8475 consensus_queue.cc:1035] T 764f2584fbfc45659d9fcdae3b0ef858 P 2fb18f58d30d4e69998fd6b5bd106bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 39557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:18:24.493172 8451 tablet_replica.cc:439] T 764f2584fbfc45659d9fcdae3b0ef858 P e94305e8231b4fd9ad7f29754b581d84: TxnStatusTablet state changed. Reason: New leader 2fb18f58d30d4e69998fd6b5bd106bff. Latest consensus state: current_term: 1 leader_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 39557 } } peers { permanent_uuid: "e94305e8231b4fd9ad7f29754b581d84" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39465 } } peers { permanent_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39501 } } }
I20260618 22:18:24.493287 8451 tablet_replica.cc:442] T 764f2584fbfc45659d9fcdae3b0ef858 P e94305e8231b4fd9ad7f29754b581d84: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 22:18:24.493237 8463 tablet_replica.cc:439] T 764f2584fbfc45659d9fcdae3b0ef858 P f85c3f5cab7b4442b6f829cfbadd33b5: TxnStatusTablet state changed. Reason: New leader 2fb18f58d30d4e69998fd6b5bd106bff. Latest consensus state: current_term: 1 leader_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 39557 } } peers { permanent_uuid: "e94305e8231b4fd9ad7f29754b581d84" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39465 } } peers { permanent_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39501 } } }
I20260618 22:18:24.493342 8463 tablet_replica.cc:442] T 764f2584fbfc45659d9fcdae3b0ef858 P f85c3f5cab7b4442b6f829cfbadd33b5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 22:18:24.495748 8475 tablet_replica.cc:439] T 764f2584fbfc45659d9fcdae3b0ef858 P 2fb18f58d30d4e69998fd6b5bd106bff: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 39557 } } peers { permanent_uuid: "e94305e8231b4fd9ad7f29754b581d84" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39465 } } peers { permanent_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39501 } } }
I20260618 22:18:24.495855 8475 tablet_replica.cc:442] T 764f2584fbfc45659d9fcdae3b0ef858 P 2fb18f58d30d4e69998fd6b5bd106bff: This TxnStatusTablet replica's current role is: LEADER
I20260618 22:18:24.497246 8463 tablet_replica.cc:439] T 764f2584fbfc45659d9fcdae3b0ef858 P f85c3f5cab7b4442b6f829cfbadd33b5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 39557 } } peers { permanent_uuid: "e94305e8231b4fd9ad7f29754b581d84" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39465 } } peers { permanent_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39501 } } }
I20260618 22:18:24.497360 8463 tablet_replica.cc:442] T 764f2584fbfc45659d9fcdae3b0ef858 P f85c3f5cab7b4442b6f829cfbadd33b5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 22:18:24.498718 8475 tablet_replica.cc:439] T 764f2584fbfc45659d9fcdae3b0ef858 P 2fb18f58d30d4e69998fd6b5bd106bff: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 39557 } } peers { permanent_uuid: "e94305e8231b4fd9ad7f29754b581d84" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39465 } } peers { permanent_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39501 } } }
I20260618 22:18:24.498813 8475 tablet_replica.cc:442] T 764f2584fbfc45659d9fcdae3b0ef858 P 2fb18f58d30d4e69998fd6b5bd106bff: This TxnStatusTablet replica's current role is: LEADER
I20260618 22:18:24.499792 8460 mvcc.cc:205] Tried to move back new op lower bound from 7298339243990114304 to 7298339243916292096. Current Snapshot: MvccSnapshot[applied={T|T < 7298339243990114304}]
I20260618 22:18:24.500005 8451 tablet_replica.cc:439] T 764f2584fbfc45659d9fcdae3b0ef858 P e94305e8231b4fd9ad7f29754b581d84: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 39557 } } peers { permanent_uuid: "e94305e8231b4fd9ad7f29754b581d84" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39465 } } peers { permanent_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39501 } } }
I20260618 22:18:24.500095 8451 tablet_replica.cc:442] T 764f2584fbfc45659d9fcdae3b0ef858 P e94305e8231b4fd9ad7f29754b581d84: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 22:18:24.504226 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:24.500928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45714 (local address 127.0.98.67:39501)
0618 22:18:24.501301 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:24.501306 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:24.501317 (+ 11us) server_negotiation.cc:407] Connection header received
0618 22:18:24.501364 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:24.501367 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:24.501407 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:24.501474 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:18:24.501705 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.502388 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.503219 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.503460 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.503570 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:24.503814 (+ 244us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:18:24.503887 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:18:24.503963 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:24.504053 (+ 90us) server_negotiation.cc:299] Negotiation successful
0618 22:18:24.504095 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311}
I20260618 22:18:24.514106 8422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:24.510510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:39501 (local address 127.0.98.67:49107)
0618 22:18:24.510642 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:24.510657 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:24.510734 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:24.510943 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:24.510946 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:24.510952 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0618 22:18:24.511052 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:24.511058 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.511673 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.511676 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:24.513003 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:24.513013 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.513415 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.513419 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:24.513510 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:24.513559 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0618 22:18:24.513805 (+ 246us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0618 22:18:24.513811 (+ 6us) client_negotiation.cc:769] Sending connection context
0618 22:18:24.513876 (+ 65us) client_negotiation.cc:240] Negotiation successful
0618 22:18:24.513971 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260618 22:18:24.514106 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:24.510544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:49107 (local address 127.0.98.67:39501)
0618 22:18:24.510678 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:24.510681 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:24.510693 (+ 12us) server_negotiation.cc:407] Connection header received
0618 22:18:24.510822 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:24.510824 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:24.510860 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:24.510919 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:18:24.511143 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.511583 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.513117 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.513317 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.513393 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:24.513675 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:18:24.513730 (+ 55us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:18:24.513781 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:24.513890 (+ 109us) server_negotiation.cc:299] Negotiation successful
0618 22:18:24.513976 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260618 22:18:24.520685 8230 raft_consensus.cc:1261] T 4185de86123b419799b2d64becd5f38a P e94305e8231b4fd9ad7f29754b581d84 [term 1 FOLLOWER]: Refusing update from remote peer 2fb18f58d30d4e69998fd6b5bd106bff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:18:24.520735 8090 raft_consensus.cc:1261] T 4185de86123b419799b2d64becd5f38a P f85c3f5cab7b4442b6f829cfbadd33b5 [term 1 FOLLOWER]: Refusing update from remote peer 2fb18f58d30d4e69998fd6b5bd106bff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:18:24.521075 8494 consensus_queue.cc:1035] T 4185de86123b419799b2d64becd5f38a P 2fb18f58d30d4e69998fd6b5bd106bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "e94305e8231b4fd9ad7f29754b581d84" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:18:24.521471 8494 consensus_queue.cc:1035] T 4185de86123b419799b2d64becd5f38a P 2fb18f58d30d4e69998fd6b5bd106bff [LEADER]: Connected to new peer: Peer: permanent_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 39557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:18:24.557287 7933 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38712:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:18:24.557449 7933 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38712:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:18:24.559599 7933 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ec613fe2b9494be494f0f98c8cfe0d63: Sending DeleteTablet for 3 replicas of tablet 4185de86123b419799b2d64becd5f38a
I20260618 22:18:24.560139 8070 tablet_service.cc:1514] Processing DeleteTablet for tablet 4185de86123b419799b2d64becd5f38a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:18:24 UTC) from {username='oryx', principal='oryx/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:58350
I20260618 22:18:24.560276 8210 tablet_service.cc:1514] Processing DeleteTablet for tablet 4185de86123b419799b2d64becd5f38a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:18:24 UTC) from {username='oryx', principal='oryx/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:36894
I20260618 22:18:24.560438 8468 tablet_replica.cc:330] T 4185de86123b419799b2d64becd5f38a P f85c3f5cab7b4442b6f829cfbadd33b5: stopping tablet replica
I20260618 22:18:24.560580 8469 tablet_replica.cc:330] T 4185de86123b419799b2d64becd5f38a P e94305e8231b4fd9ad7f29754b581d84: stopping tablet replica
I20260618 22:18:24.560582 8468 raft_consensus.cc:2229] T 4185de86123b419799b2d64becd5f38a P f85c3f5cab7b4442b6f829cfbadd33b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:18:24.560680 8468 raft_consensus.cc:2258] T 4185de86123b419799b2d64becd5f38a P f85c3f5cab7b4442b6f829cfbadd33b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:18:24.560699 8469 raft_consensus.cc:2229] T 4185de86123b419799b2d64becd5f38a P e94305e8231b4fd9ad7f29754b581d84 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:18:24.560832 8469 raft_consensus.cc:2258] T 4185de86123b419799b2d64becd5f38a P e94305e8231b4fd9ad7f29754b581d84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:18:24.561017 8349 tablet_service.cc:1514] Processing DeleteTablet for tablet 4185de86123b419799b2d64becd5f38a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:18:24 UTC) from {username='oryx', principal='oryx/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:45698
I20260618 22:18:24.561291 8470 tablet_replica.cc:330] T 4185de86123b419799b2d64becd5f38a P 2fb18f58d30d4e69998fd6b5bd106bff: stopping tablet replica
I20260618 22:18:24.561424 8470 raft_consensus.cc:2229] T 4185de86123b419799b2d64becd5f38a P 2fb18f58d30d4e69998fd6b5bd106bff [term 1 LEADER]: Raft consensus shutting down.
I20260618 22:18:24.561597 8468 ts_tablet_manager.cc:1905] T 4185de86123b419799b2d64becd5f38a P f85c3f5cab7b4442b6f829cfbadd33b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:18:24.561646 8470 raft_consensus.cc:2258] T 4185de86123b419799b2d64becd5f38a P 2fb18f58d30d4e69998fd6b5bd106bff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:18:24.562443 8470 ts_tablet_manager.cc:1905] T 4185de86123b419799b2d64becd5f38a P 2fb18f58d30d4e69998fd6b5bd106bff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:18:24.564458 8468 ts_tablet_manager.cc:1918] T 4185de86123b419799b2d64becd5f38a P f85c3f5cab7b4442b6f829cfbadd33b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260618 22:18:24.564525 8468 log.cc:1196] T 4185de86123b419799b2d64becd5f38a P f85c3f5cab7b4442b6f829cfbadd33b5: Deleting WAL directory at /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/ts-0/wal/wals/4185de86123b419799b2d64becd5f38a
I20260618 22:18:24.564769 8468 ts_tablet_manager.cc:1939] T 4185de86123b419799b2d64becd5f38a P f85c3f5cab7b4442b6f829cfbadd33b5: Deleting consensus metadata
I20260618 22:18:24.565089 8470 ts_tablet_manager.cc:1918] T 4185de86123b419799b2d64becd5f38a P 2fb18f58d30d4e69998fd6b5bd106bff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260618 22:18:24.565140 8470 log.cc:1196] T 4185de86123b419799b2d64becd5f38a P 2fb18f58d30d4e69998fd6b5bd106bff: Deleting WAL directory at /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/ts-2/wal/wals/4185de86123b419799b2d64becd5f38a
I20260618 22:18:24.565389 8470 ts_tablet_manager.cc:1939] T 4185de86123b419799b2d64becd5f38a P 2fb18f58d30d4e69998fd6b5bd106bff: Deleting consensus metadata
I20260618 22:18:24.565449 7918 catalog_manager.cc:4841] TS f85c3f5cab7b4442b6f829cfbadd33b5 (127.0.98.65:39557): tablet 4185de86123b419799b2d64becd5f38a (table test-table [id=148d54c0c51e49c5bbf7f567dcebacd2]) successfully deleted
I20260618 22:18:24.566068 7919 catalog_manager.cc:4841] TS 2fb18f58d30d4e69998fd6b5bd106bff (127.0.98.67:39501): tablet 4185de86123b419799b2d64becd5f38a (table test-table [id=148d54c0c51e49c5bbf7f567dcebacd2]) successfully deleted
I20260618 22:18:24.566967 8469 ts_tablet_manager.cc:1905] T 4185de86123b419799b2d64becd5f38a P e94305e8231b4fd9ad7f29754b581d84: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:18:24.569087 8469 ts_tablet_manager.cc:1918] T 4185de86123b419799b2d64becd5f38a P e94305e8231b4fd9ad7f29754b581d84: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260618 22:18:24.569149 8469 log.cc:1196] T 4185de86123b419799b2d64becd5f38a P e94305e8231b4fd9ad7f29754b581d84: Deleting WAL directory at /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/ts-1/wal/wals/4185de86123b419799b2d64becd5f38a
I20260618 22:18:24.569391 8469 ts_tablet_manager.cc:1939] T 4185de86123b419799b2d64becd5f38a P e94305e8231b4fd9ad7f29754b581d84: Deleting consensus metadata
I20260618 22:18:24.570024 7919 catalog_manager.cc:4841] TS e94305e8231b4fd9ad7f29754b581d84 (127.0.98.66:39465): tablet 4185de86123b419799b2d64becd5f38a (table test-table [id=148d54c0c51e49c5bbf7f567dcebacd2]) successfully deleted
Jun 18 22:18:24 dist-test-slave-g6r4 krb5kdc[7888](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821104, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 18 22:18:24 dist-test-slave-g6r4 krb5kdc[7888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:18:24 dist-test-slave-g6r4 krb5kdc[7888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
W20260618 22:18:24.600953 7982 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:18:24.593153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38718 (local address 127.0.98.126:37239)
0618 22:18:24.593271 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:24.593276 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:24.593607 (+ 331us) server_negotiation.cc:407] Connection header received
0618 22:18:24.593802 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:24.593806 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:24.593867 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:24.594012 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:24.594820 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.595660 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.596864 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.597217 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.597335 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:24.600833 (+ 3498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38718: BlockingRecv error: recv got EOF from 127.0.0.1:38718 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 18 22:18:24 dist-test-slave-g6r4 krb5kdc[7888](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821104, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM
I20260618 22:18:24.612453 7982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:24.602880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38726 (local address 127.0.98.126:37239)
0618 22:18:24.602982 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:24.602986 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:24.603167 (+ 181us) server_negotiation.cc:407] Connection header received
0618 22:18:24.603251 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:24.603254 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:24.603302 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:24.603396 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:24.604049 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.604871 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.606009 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.606316 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.606440 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:24.609147 (+ 2707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:24.609150 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:24.609153 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:24.609173 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:24.610753 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:24.611232 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:24.611235 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:24.611237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:24.611315 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:24.611611 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:24.611614 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:24.611615 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:24.611830 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:24.611938 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:24.612198 (+ 260us) server_negotiation.cc:299] Negotiation successful
0618 22:18:24.612288 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260618 22:18:24.615338 7933 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38726:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 22:18:24.615774 7933 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:18:24.620476 8070 tablet_service.cc:1467] Processing CreateTablet for tablet 873f5ca25bbb4f79b01cc2d5360046fa (DEFAULT_TABLE table=test-table [id=5a99b8e423d645118085408596d9345a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:18:24.620767 8070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 873f5ca25bbb4f79b01cc2d5360046fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:18:24.620726 8349 tablet_service.cc:1467] Processing CreateTablet for tablet 873f5ca25bbb4f79b01cc2d5360046fa (DEFAULT_TABLE table=test-table [id=5a99b8e423d645118085408596d9345a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:18:24.620695 8210 tablet_service.cc:1467] Processing CreateTablet for tablet 873f5ca25bbb4f79b01cc2d5360046fa (DEFAULT_TABLE table=test-table [id=5a99b8e423d645118085408596d9345a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:18:24.620991 8210 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 873f5ca25bbb4f79b01cc2d5360046fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:18:24.621017 8349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 873f5ca25bbb4f79b01cc2d5360046fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:18:24.623500 8446 tablet_bootstrap.cc:492] T 873f5ca25bbb4f79b01cc2d5360046fa P e94305e8231b4fd9ad7f29754b581d84: Bootstrap starting.
I20260618 22:18:24.623826 8444 tablet_bootstrap.cc:492] T 873f5ca25bbb4f79b01cc2d5360046fa P 2fb18f58d30d4e69998fd6b5bd106bff: Bootstrap starting.
I20260618 22:18:24.624502 8446 tablet_bootstrap.cc:654] T 873f5ca25bbb4f79b01cc2d5360046fa P e94305e8231b4fd9ad7f29754b581d84: Neither blocks nor log segments found. Creating new log.
I20260618 22:18:24.625073 8444 tablet_bootstrap.cc:654] T 873f5ca25bbb4f79b01cc2d5360046fa P 2fb18f58d30d4e69998fd6b5bd106bff: Neither blocks nor log segments found. Creating new log.
I20260618 22:18:24.625105 8445 tablet_bootstrap.cc:492] T 873f5ca25bbb4f79b01cc2d5360046fa P f85c3f5cab7b4442b6f829cfbadd33b5: Bootstrap starting.
I20260618 22:18:24.625934 8445 tablet_bootstrap.cc:654] T 873f5ca25bbb4f79b01cc2d5360046fa P f85c3f5cab7b4442b6f829cfbadd33b5: Neither blocks nor log segments found. Creating new log.
I20260618 22:18:24.626221 8444 tablet_bootstrap.cc:492] T 873f5ca25bbb4f79b01cc2d5360046fa P 2fb18f58d30d4e69998fd6b5bd106bff: No bootstrap required, opened a new log
I20260618 22:18:24.626295 8444 ts_tablet_manager.cc:1397] T 873f5ca25bbb4f79b01cc2d5360046fa P 2fb18f58d30d4e69998fd6b5bd106bff: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260618 22:18:24.626784 8444 raft_consensus.cc:348] T 873f5ca25bbb4f79b01cc2d5360046fa P 2fb18f58d30d4e69998fd6b5bd106bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 39557 } } peers { permanent_uuid: "e94305e8231b4fd9ad7f29754b581d84" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39465 } } peers { permanent_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39501 } }
I20260618 22:18:24.626917 8444 raft_consensus.cc:374] T 873f5ca25bbb4f79b01cc2d5360046fa P 2fb18f58d30d4e69998fd6b5bd106bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:18:24.626955 8444 raft_consensus.cc:729] T 873f5ca25bbb4f79b01cc2d5360046fa P 2fb18f58d30d4e69998fd6b5bd106bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fb18f58d30d4e69998fd6b5bd106bff, State: Initialized, Role: FOLLOWER
I20260618 22:18:24.626979 8446 tablet_bootstrap.cc:492] T 873f5ca25bbb4f79b01cc2d5360046fa P e94305e8231b4fd9ad7f29754b581d84: No bootstrap required, opened a new log
I20260618 22:18:24.627050 8446 ts_tablet_manager.cc:1397] T 873f5ca25bbb4f79b01cc2d5360046fa P e94305e8231b4fd9ad7f29754b581d84: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260618 22:18:24.627079 8444 consensus_queue.cc:260] T 873f5ca25bbb4f79b01cc2d5360046fa P 2fb18f58d30d4e69998fd6b5bd106bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 39557 } } peers { permanent_uuid: "e94305e8231b4fd9ad7f29754b581d84" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39465 } } peers { permanent_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39501 } }
I20260618 22:18:24.627408 8444 ts_tablet_manager.cc:1428] T 873f5ca25bbb4f79b01cc2d5360046fa P 2fb18f58d30d4e69998fd6b5bd106bff: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260618 22:18:24.627480 8446 raft_consensus.cc:348] T 873f5ca25bbb4f79b01cc2d5360046fa P e94305e8231b4fd9ad7f29754b581d84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 39557 } } peers { permanent_uuid: "e94305e8231b4fd9ad7f29754b581d84" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39465 } } peers { permanent_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39501 } }
I20260618 22:18:24.627578 8446 raft_consensus.cc:374] T 873f5ca25bbb4f79b01cc2d5360046fa P e94305e8231b4fd9ad7f29754b581d84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:18:24.627609 8446 raft_consensus.cc:729] T 873f5ca25bbb4f79b01cc2d5360046fa P e94305e8231b4fd9ad7f29754b581d84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e94305e8231b4fd9ad7f29754b581d84, State: Initialized, Role: FOLLOWER
I20260618 22:18:24.627717 8446 consensus_queue.cc:260] T 873f5ca25bbb4f79b01cc2d5360046fa P e94305e8231b4fd9ad7f29754b581d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 39557 } } peers { permanent_uuid: "e94305e8231b4fd9ad7f29754b581d84" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39465 } } peers { permanent_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39501 } }
I20260618 22:18:24.627975 8446 ts_tablet_manager.cc:1428] T 873f5ca25bbb4f79b01cc2d5360046fa P e94305e8231b4fd9ad7f29754b581d84: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260618 22:18:24.628741 8445 tablet_bootstrap.cc:492] T 873f5ca25bbb4f79b01cc2d5360046fa P f85c3f5cab7b4442b6f829cfbadd33b5: No bootstrap required, opened a new log
I20260618 22:18:24.628815 8445 ts_tablet_manager.cc:1397] T 873f5ca25bbb4f79b01cc2d5360046fa P f85c3f5cab7b4442b6f829cfbadd33b5: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260618 22:18:24.629192 8445 raft_consensus.cc:348] T 873f5ca25bbb4f79b01cc2d5360046fa P f85c3f5cab7b4442b6f829cfbadd33b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 39557 } } peers { permanent_uuid: "e94305e8231b4fd9ad7f29754b581d84" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39465 } } peers { permanent_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39501 } }
I20260618 22:18:24.629338 8445 raft_consensus.cc:374] T 873f5ca25bbb4f79b01cc2d5360046fa P f85c3f5cab7b4442b6f829cfbadd33b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:18:24.629379 8445 raft_consensus.cc:729] T 873f5ca25bbb4f79b01cc2d5360046fa P f85c3f5cab7b4442b6f829cfbadd33b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f85c3f5cab7b4442b6f829cfbadd33b5, State: Initialized, Role: FOLLOWER
I20260618 22:18:24.629530 8445 consensus_queue.cc:260] T 873f5ca25bbb4f79b01cc2d5360046fa P f85c3f5cab7b4442b6f829cfbadd33b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 39557 } } peers { permanent_uuid: "e94305e8231b4fd9ad7f29754b581d84" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39465 } } peers { permanent_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39501 } }
I20260618 22:18:24.629823 8445 ts_tablet_manager.cc:1428] T 873f5ca25bbb4f79b01cc2d5360046fa P f85c3f5cab7b4442b6f829cfbadd33b5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260618 22:18:24.698176 8463 raft_consensus.cc:482] T 873f5ca25bbb4f79b01cc2d5360046fa P f85c3f5cab7b4442b6f829cfbadd33b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:18:24.698356 8463 raft_consensus.cc:504] T 873f5ca25bbb4f79b01cc2d5360046fa P f85c3f5cab7b4442b6f829cfbadd33b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 39557 } } peers { permanent_uuid: "e94305e8231b4fd9ad7f29754b581d84" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39465 } } peers { permanent_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39501 } }
I20260618 22:18:24.698849 8463 leader_election.cc:288] T 873f5ca25bbb4f79b01cc2d5360046fa P f85c3f5cab7b4442b6f829cfbadd33b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e94305e8231b4fd9ad7f29754b581d84 (127.0.98.66:39465), 2fb18f58d30d4e69998fd6b5bd106bff (127.0.98.67:39501)
I20260618 22:18:24.699380 8370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "873f5ca25bbb4f79b01cc2d5360046fa" candidate_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" is_pre_election: true
I20260618 22:18:24.699380 8230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "873f5ca25bbb4f79b01cc2d5360046fa" candidate_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e94305e8231b4fd9ad7f29754b581d84" is_pre_election: true
I20260618 22:18:24.699534 8370 raft_consensus.cc:2454] T 873f5ca25bbb4f79b01cc2d5360046fa P 2fb18f58d30d4e69998fd6b5bd106bff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f85c3f5cab7b4442b6f829cfbadd33b5 in term 0.
I20260618 22:18:24.699568 8230 raft_consensus.cc:2454] T 873f5ca25bbb4f79b01cc2d5360046fa P e94305e8231b4fd9ad7f29754b581d84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f85c3f5cab7b4442b6f829cfbadd33b5 in term 0.
I20260618 22:18:24.699858 8023 leader_election.cc:302] T 873f5ca25bbb4f79b01cc2d5360046fa P f85c3f5cab7b4442b6f829cfbadd33b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fb18f58d30d4e69998fd6b5bd106bff, f85c3f5cab7b4442b6f829cfbadd33b5; no voters:
I20260618 22:18:24.700060 8463 raft_consensus.cc:2781] T 873f5ca25bbb4f79b01cc2d5360046fa P f85c3f5cab7b4442b6f829cfbadd33b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:18:24.700124 8463 raft_consensus.cc:482] T 873f5ca25bbb4f79b01cc2d5360046fa P f85c3f5cab7b4442b6f829cfbadd33b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:18:24.700160 8463 raft_consensus.cc:3037] T 873f5ca25bbb4f79b01cc2d5360046fa P f85c3f5cab7b4442b6f829cfbadd33b5 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:18:24.701391 8463 raft_consensus.cc:504] T 873f5ca25bbb4f79b01cc2d5360046fa P f85c3f5cab7b4442b6f829cfbadd33b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 39557 } } peers { permanent_uuid: "e94305e8231b4fd9ad7f29754b581d84" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39465 } } peers { permanent_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39501 } }
I20260618 22:18:24.701738 8463 leader_election.cc:288] T 873f5ca25bbb4f79b01cc2d5360046fa P f85c3f5cab7b4442b6f829cfbadd33b5 [CANDIDATE]: Term 1 election: Requested vote from peers e94305e8231b4fd9ad7f29754b581d84 (127.0.98.66:39465), 2fb18f58d30d4e69998fd6b5bd106bff (127.0.98.67:39501)
I20260618 22:18:24.702109 8230 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "873f5ca25bbb4f79b01cc2d5360046fa" candidate_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e94305e8231b4fd9ad7f29754b581d84"
I20260618 22:18:24.702226 8230 raft_consensus.cc:3037] T 873f5ca25bbb4f79b01cc2d5360046fa P e94305e8231b4fd9ad7f29754b581d84 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:18:24.703280 8230 raft_consensus.cc:2454] T 873f5ca25bbb4f79b01cc2d5360046fa P e94305e8231b4fd9ad7f29754b581d84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f85c3f5cab7b4442b6f829cfbadd33b5 in term 1.
I20260618 22:18:24.703516 8370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "873f5ca25bbb4f79b01cc2d5360046fa" candidate_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fb18f58d30d4e69998fd6b5bd106bff"
I20260618 22:18:24.703629 8370 raft_consensus.cc:3037] T 873f5ca25bbb4f79b01cc2d5360046fa P 2fb18f58d30d4e69998fd6b5bd106bff [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:18:24.704648 8370 raft_consensus.cc:2454] T 873f5ca25bbb4f79b01cc2d5360046fa P 2fb18f58d30d4e69998fd6b5bd106bff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f85c3f5cab7b4442b6f829cfbadd33b5 in term 1.
I20260618 22:18:24.704926 8023 leader_election.cc:302] T 873f5ca25bbb4f79b01cc2d5360046fa P f85c3f5cab7b4442b6f829cfbadd33b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fb18f58d30d4e69998fd6b5bd106bff, f85c3f5cab7b4442b6f829cfbadd33b5; no voters:
I20260618 22:18:24.705091 8463 raft_consensus.cc:2781] T 873f5ca25bbb4f79b01cc2d5360046fa P f85c3f5cab7b4442b6f829cfbadd33b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:18:24.705255 8463 raft_consensus.cc:686] T 873f5ca25bbb4f79b01cc2d5360046fa P f85c3f5cab7b4442b6f829cfbadd33b5 [term 1 LEADER]: Becoming Leader. State: Replica: f85c3f5cab7b4442b6f829cfbadd33b5, State: Running, Role: LEADER
I20260618 22:18:24.705397 8463 consensus_queue.cc:237] T 873f5ca25bbb4f79b01cc2d5360046fa P f85c3f5cab7b4442b6f829cfbadd33b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 39557 } } peers { permanent_uuid: "e94305e8231b4fd9ad7f29754b581d84" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39465 } } peers { permanent_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39501 } }
I20260618 22:18:24.707017 7932 catalog_manager.cc:5495] T 873f5ca25bbb4f79b01cc2d5360046fa P f85c3f5cab7b4442b6f829cfbadd33b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to f85c3f5cab7b4442b6f829cfbadd33b5 (127.0.98.65). New cstate: current_term: 1 leader_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 39557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e94305e8231b4fd9ad7f29754b581d84" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39465 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39501 } health_report { overall_health: UNKNOWN } } }
I20260618 22:18:24.727347 8439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:24.723952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58364 (local address 127.0.98.65:39557)
0618 22:18:24.724089 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:24.724093 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:24.724111 (+ 18us) server_negotiation.cc:407] Connection header received
0618 22:18:24.724245 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:24.724247 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:24.724303 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:24.724374 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:18:24.724634 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.725153 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.726375 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.726581 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.726652 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:24.726919 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:18:24.726979 (+ 60us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:18:24.727029 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:24.727146 (+ 117us) server_negotiation.cc:299] Negotiation successful
0618 22:18:24.727190 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260618 22:18:24.729194 8230 raft_consensus.cc:1261] T 873f5ca25bbb4f79b01cc2d5360046fa P e94305e8231b4fd9ad7f29754b581d84 [term 1 FOLLOWER]: Refusing update from remote peer f85c3f5cab7b4442b6f829cfbadd33b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:18:24.729239 8370 raft_consensus.cc:1261] T 873f5ca25bbb4f79b01cc2d5360046fa P 2fb18f58d30d4e69998fd6b5bd106bff [term 1 FOLLOWER]: Refusing update from remote peer f85c3f5cab7b4442b6f829cfbadd33b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:18:24.729605 8463 consensus_queue.cc:1035] T 873f5ca25bbb4f79b01cc2d5360046fa P f85c3f5cab7b4442b6f829cfbadd33b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:18:24.729734 8513 consensus_queue.cc:1035] T 873f5ca25bbb4f79b01cc2d5360046fa P f85c3f5cab7b4442b6f829cfbadd33b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e94305e8231b4fd9ad7f29754b581d84" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39465 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:18:24.737178 7932 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:18:24.737311 7932 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:18:24.739804 7932 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ec613fe2b9494be494f0f98c8cfe0d63: Sending DeleteTablet for 3 replicas of tablet 873f5ca25bbb4f79b01cc2d5360046fa
I20260618 22:18:24.740464 8210 tablet_service.cc:1514] Processing DeleteTablet for tablet 873f5ca25bbb4f79b01cc2d5360046fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:18:24 UTC) from {username='oryx', principal='oryx/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:36894
I20260618 22:18:24.740513 8349 tablet_service.cc:1514] Processing DeleteTablet for tablet 873f5ca25bbb4f79b01cc2d5360046fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:18:24 UTC) from {username='oryx', principal='oryx/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:45698
I20260618 22:18:24.740464 8209 tablet_service.cc:1514] Processing DeleteTablet for tablet 873f5ca25bbb4f79b01cc2d5360046fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:18:24 UTC) from {username='oryx', principal='oryx/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:36894
I20260618 22:18:24.740666 8470 tablet_replica.cc:330] T 873f5ca25bbb4f79b01cc2d5360046fa P 2fb18f58d30d4e69998fd6b5bd106bff: stopping tablet replica
I20260618 22:18:24.740674 8469 tablet_replica.cc:330] T 873f5ca25bbb4f79b01cc2d5360046fa P e94305e8231b4fd9ad7f29754b581d84: stopping tablet replica
I20260618 22:18:24.740772 8470 raft_consensus.cc:2229] T 873f5ca25bbb4f79b01cc2d5360046fa P 2fb18f58d30d4e69998fd6b5bd106bff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:18:24.740779 8469 raft_consensus.cc:2229] T 873f5ca25bbb4f79b01cc2d5360046fa P e94305e8231b4fd9ad7f29754b581d84 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:18:24.740880 8470 raft_consensus.cc:2258] T 873f5ca25bbb4f79b01cc2d5360046fa P 2fb18f58d30d4e69998fd6b5bd106bff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:18:24.740913 8469 raft_consensus.cc:2258] T 873f5ca25bbb4f79b01cc2d5360046fa P e94305e8231b4fd9ad7f29754b581d84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:18:24.741698 8469 ts_tablet_manager.cc:1905] T 873f5ca25bbb4f79b01cc2d5360046fa P e94305e8231b4fd9ad7f29754b581d84: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:18:24.741816 8470 ts_tablet_manager.cc:1905] T 873f5ca25bbb4f79b01cc2d5360046fa P 2fb18f58d30d4e69998fd6b5bd106bff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:18:24.742231 8070 tablet_service.cc:1514] Processing DeleteTablet for tablet 873f5ca25bbb4f79b01cc2d5360046fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:18:24 UTC) from {username='oryx', principal='oryx/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:58350
I20260618 22:18:24.742393 8468 tablet_replica.cc:330] T 873f5ca25bbb4f79b01cc2d5360046fa P f85c3f5cab7b4442b6f829cfbadd33b5: stopping tablet replica
I20260618 22:18:24.742483 8468 raft_consensus.cc:2229] T 873f5ca25bbb4f79b01cc2d5360046fa P f85c3f5cab7b4442b6f829cfbadd33b5 [term 1 LEADER]: Raft consensus shutting down.
I20260618 22:18:24.742688 8468 raft_consensus.cc:2258] T 873f5ca25bbb4f79b01cc2d5360046fa P f85c3f5cab7b4442b6f829cfbadd33b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:18:24.744181 8469 ts_tablet_manager.cc:1918] T 873f5ca25bbb4f79b01cc2d5360046fa P e94305e8231b4fd9ad7f29754b581d84: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:18:24.744258 8469 log.cc:1196] T 873f5ca25bbb4f79b01cc2d5360046fa P e94305e8231b4fd9ad7f29754b581d84: Deleting WAL directory at /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/ts-1/wal/wals/873f5ca25bbb4f79b01cc2d5360046fa
I20260618 22:18:24.744494 8469 ts_tablet_manager.cc:1939] T 873f5ca25bbb4f79b01cc2d5360046fa P e94305e8231b4fd9ad7f29754b581d84: Deleting consensus metadata
I20260618 22:18:24.745046 8470 ts_tablet_manager.cc:1918] T 873f5ca25bbb4f79b01cc2d5360046fa P 2fb18f58d30d4e69998fd6b5bd106bff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:18:24.745110 8470 log.cc:1196] T 873f5ca25bbb4f79b01cc2d5360046fa P 2fb18f58d30d4e69998fd6b5bd106bff: Deleting WAL directory at /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/ts-2/wal/wals/873f5ca25bbb4f79b01cc2d5360046fa
I20260618 22:18:24.745167 7919 catalog_manager.cc:4841] TS e94305e8231b4fd9ad7f29754b581d84 (127.0.98.66:39465): tablet 873f5ca25bbb4f79b01cc2d5360046fa (table test-table [id=5a99b8e423d645118085408596d9345a]) successfully deleted
I20260618 22:18:24.745357 8470 ts_tablet_manager.cc:1939] T 873f5ca25bbb4f79b01cc2d5360046fa P 2fb18f58d30d4e69998fd6b5bd106bff: Deleting consensus metadata
W20260618 22:18:24.745442 7919 catalog_manager.cc:4808] TS e94305e8231b4fd9ad7f29754b581d84 (127.0.98.66:39465): delete failed for tablet 873f5ca25bbb4f79b01cc2d5360046fa because the tablet was not found. No further retry: Not found: Tablet not found: 873f5ca25bbb4f79b01cc2d5360046fa
I20260618 22:18:24.746006 8468 ts_tablet_manager.cc:1905] T 873f5ca25bbb4f79b01cc2d5360046fa P f85c3f5cab7b4442b6f829cfbadd33b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:18:24.746032 7919 catalog_manager.cc:4841] TS 2fb18f58d30d4e69998fd6b5bd106bff (127.0.98.67:39501): tablet 873f5ca25bbb4f79b01cc2d5360046fa (table test-table [id=5a99b8e423d645118085408596d9345a]) successfully deleted
I20260618 22:18:24.746201 7931 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38726:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 22:18:24.746578 7931 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:18:24.748289 8468 ts_tablet_manager.cc:1918] T 873f5ca25bbb4f79b01cc2d5360046fa P f85c3f5cab7b4442b6f829cfbadd33b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:18:24.748360 8468 log.cc:1196] T 873f5ca25bbb4f79b01cc2d5360046fa P f85c3f5cab7b4442b6f829cfbadd33b5: Deleting WAL directory at /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/ts-0/wal/wals/873f5ca25bbb4f79b01cc2d5360046fa
I20260618 22:18:24.748606 8468 ts_tablet_manager.cc:1939] T 873f5ca25bbb4f79b01cc2d5360046fa P f85c3f5cab7b4442b6f829cfbadd33b5: Deleting consensus metadata
I20260618 22:18:24.749229 7918 catalog_manager.cc:4841] TS f85c3f5cab7b4442b6f829cfbadd33b5 (127.0.98.65:39557): tablet 873f5ca25bbb4f79b01cc2d5360046fa (table test-table [id=5a99b8e423d645118085408596d9345a]) successfully deleted
I20260618 22:18:24.751413 8209 tablet_service.cc:1467] Processing CreateTablet for tablet 48a3797d6c7444708666c32a3783f457 (DEFAULT_TABLE table=test-table [id=35a7bdaea358412096e98521aed2c102]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:18:24.751482 8070 tablet_service.cc:1467] Processing CreateTablet for tablet 48a3797d6c7444708666c32a3783f457 (DEFAULT_TABLE table=test-table [id=35a7bdaea358412096e98521aed2c102]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:18:24.751684 8209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48a3797d6c7444708666c32a3783f457. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:18:24.751730 8070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48a3797d6c7444708666c32a3783f457. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:18:24.753808 8446 tablet_bootstrap.cc:492] T 48a3797d6c7444708666c32a3783f457 P e94305e8231b4fd9ad7f29754b581d84: Bootstrap starting.
I20260618 22:18:24.753844 8445 tablet_bootstrap.cc:492] T 48a3797d6c7444708666c32a3783f457 P f85c3f5cab7b4442b6f829cfbadd33b5: Bootstrap starting.
I20260618 22:18:24.754568 8446 tablet_bootstrap.cc:654] T 48a3797d6c7444708666c32a3783f457 P e94305e8231b4fd9ad7f29754b581d84: Neither blocks nor log segments found. Creating new log.
I20260618 22:18:24.754650 8445 tablet_bootstrap.cc:654] T 48a3797d6c7444708666c32a3783f457 P f85c3f5cab7b4442b6f829cfbadd33b5: Neither blocks nor log segments found. Creating new log.
I20260618 22:18:24.755229 8349 tablet_service.cc:1467] Processing CreateTablet for tablet 48a3797d6c7444708666c32a3783f457 (DEFAULT_TABLE table=test-table [id=35a7bdaea358412096e98521aed2c102]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:18:24.755492 8349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48a3797d6c7444708666c32a3783f457. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:18:24.755760 8446 tablet_bootstrap.cc:492] T 48a3797d6c7444708666c32a3783f457 P e94305e8231b4fd9ad7f29754b581d84: No bootstrap required, opened a new log
I20260618 22:18:24.755857 8446 ts_tablet_manager.cc:1397] T 48a3797d6c7444708666c32a3783f457 P e94305e8231b4fd9ad7f29754b581d84: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260618 22:18:24.756237 8446 raft_consensus.cc:348] T 48a3797d6c7444708666c32a3783f457 P e94305e8231b4fd9ad7f29754b581d84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 39557 } } peers { permanent_uuid: "e94305e8231b4fd9ad7f29754b581d84" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39465 } } peers { permanent_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39501 } }
I20260618 22:18:24.756345 8446 raft_consensus.cc:374] T 48a3797d6c7444708666c32a3783f457 P e94305e8231b4fd9ad7f29754b581d84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:18:24.756385 8446 raft_consensus.cc:729] T 48a3797d6c7444708666c32a3783f457 P e94305e8231b4fd9ad7f29754b581d84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e94305e8231b4fd9ad7f29754b581d84, State: Initialized, Role: FOLLOWER
I20260618 22:18:24.756517 8446 consensus_queue.cc:260] T 48a3797d6c7444708666c32a3783f457 P e94305e8231b4fd9ad7f29754b581d84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 39557 } } peers { permanent_uuid: "e94305e8231b4fd9ad7f29754b581d84" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39465 } } peers { permanent_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39501 } }
I20260618 22:18:24.756780 8446 ts_tablet_manager.cc:1428] T 48a3797d6c7444708666c32a3783f457 P e94305e8231b4fd9ad7f29754b581d84: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260618 22:18:24.757973 8444 tablet_bootstrap.cc:492] T 48a3797d6c7444708666c32a3783f457 P 2fb18f58d30d4e69998fd6b5bd106bff: Bootstrap starting.
I20260618 22:18:24.758764 8444 tablet_bootstrap.cc:654] T 48a3797d6c7444708666c32a3783f457 P 2fb18f58d30d4e69998fd6b5bd106bff: Neither blocks nor log segments found. Creating new log.
I20260618 22:18:24.760025 8445 tablet_bootstrap.cc:492] T 48a3797d6c7444708666c32a3783f457 P f85c3f5cab7b4442b6f829cfbadd33b5: No bootstrap required, opened a new log
I20260618 22:18:24.760032 8444 tablet_bootstrap.cc:492] T 48a3797d6c7444708666c32a3783f457 P 2fb18f58d30d4e69998fd6b5bd106bff: No bootstrap required, opened a new log
I20260618 22:18:24.760119 8444 ts_tablet_manager.cc:1397] T 48a3797d6c7444708666c32a3783f457 P 2fb18f58d30d4e69998fd6b5bd106bff: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260618 22:18:24.760123 8445 ts_tablet_manager.cc:1397] T 48a3797d6c7444708666c32a3783f457 P f85c3f5cab7b4442b6f829cfbadd33b5: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260618 22:18:24.760550 8445 raft_consensus.cc:348] T 48a3797d6c7444708666c32a3783f457 P f85c3f5cab7b4442b6f829cfbadd33b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 39557 } } peers { permanent_uuid: "e94305e8231b4fd9ad7f29754b581d84" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39465 } } peers { permanent_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39501 } }
I20260618 22:18:24.760560 8444 raft_consensus.cc:348] T 48a3797d6c7444708666c32a3783f457 P 2fb18f58d30d4e69998fd6b5bd106bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 39557 } } peers { permanent_uuid: "e94305e8231b4fd9ad7f29754b581d84" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39465 } } peers { permanent_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39501 } }
I20260618 22:18:24.760660 8445 raft_consensus.cc:374] T 48a3797d6c7444708666c32a3783f457 P f85c3f5cab7b4442b6f829cfbadd33b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:18:24.760684 8444 raft_consensus.cc:374] T 48a3797d6c7444708666c32a3783f457 P 2fb18f58d30d4e69998fd6b5bd106bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:18:24.760700 8445 raft_consensus.cc:729] T 48a3797d6c7444708666c32a3783f457 P f85c3f5cab7b4442b6f829cfbadd33b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f85c3f5cab7b4442b6f829cfbadd33b5, State: Initialized, Role: FOLLOWER
I20260618 22:18:24.760721 8444 raft_consensus.cc:729] T 48a3797d6c7444708666c32a3783f457 P 2fb18f58d30d4e69998fd6b5bd106bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fb18f58d30d4e69998fd6b5bd106bff, State: Initialized, Role: FOLLOWER
I20260618 22:18:24.760821 8445 consensus_queue.cc:260] T 48a3797d6c7444708666c32a3783f457 P f85c3f5cab7b4442b6f829cfbadd33b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 39557 } } peers { permanent_uuid: "e94305e8231b4fd9ad7f29754b581d84" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39465 } } peers { permanent_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39501 } }
I20260618 22:18:24.760843 8444 consensus_queue.cc:260] T 48a3797d6c7444708666c32a3783f457 P 2fb18f58d30d4e69998fd6b5bd106bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 39557 } } peers { permanent_uuid: "e94305e8231b4fd9ad7f29754b581d84" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39465 } } peers { permanent_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39501 } }
I20260618 22:18:24.761091 8445 ts_tablet_manager.cc:1428] T 48a3797d6c7444708666c32a3783f457 P f85c3f5cab7b4442b6f829cfbadd33b5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260618 22:18:24.761099 8444 ts_tablet_manager.cc:1428] T 48a3797d6c7444708666c32a3783f457 P 2fb18f58d30d4e69998fd6b5bd106bff: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260618 22:18:24.883553 8451 raft_consensus.cc:482] T 48a3797d6c7444708666c32a3783f457 P e94305e8231b4fd9ad7f29754b581d84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:18:24.883700 8451 raft_consensus.cc:504] T 48a3797d6c7444708666c32a3783f457 P e94305e8231b4fd9ad7f29754b581d84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 39557 } } peers { permanent_uuid: "e94305e8231b4fd9ad7f29754b581d84" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39465 } } peers { permanent_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39501 } }
I20260618 22:18:24.884977 8451 leader_election.cc:288] T 48a3797d6c7444708666c32a3783f457 P e94305e8231b4fd9ad7f29754b581d84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f85c3f5cab7b4442b6f829cfbadd33b5 (127.0.98.65:39557), 2fb18f58d30d4e69998fd6b5bd106bff (127.0.98.67:39501)
I20260618 22:18:24.890465 8439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:24.885278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:39995 (local address 127.0.98.65:39557)
0618 22:18:24.885465 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:24.885470 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:24.885541 (+ 71us) server_negotiation.cc:407] Connection header received
0618 22:18:24.885975 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:24.885978 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:24.886027 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:24.886102 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:18:24.886329 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.887146 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.889001 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.889656 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.889774 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:24.889848 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:24.890265 (+ 417us) server_negotiation.cc:299] Negotiation successful
0618 22:18:24.890334 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260618 22:18:24.890465 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:24.885326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:39459 (local address 127.0.98.67:39501)
0618 22:18:24.885487 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:24.885493 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:24.885612 (+ 119us) server_negotiation.cc:407] Connection header received
0618 22:18:24.885805 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:24.885808 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:24.885851 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:24.885999 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:18:24.886329 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.887146 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.889001 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.889656 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.889817 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:24.889875 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:24.890255 (+ 380us) server_negotiation.cc:299] Negotiation successful
0618 22:18:24.890329 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260618 22:18:24.890795 8521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:24.885205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:39501 (local address 127.0.98.66:39459)
0618 22:18:24.885526 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:24.885539 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:24.885642 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:24.886031 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:24.886034 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:24.886051 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:18:24.886211 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:24.886218 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.887290 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.887295 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:24.888871 (+ 1576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:24.888878 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.889826 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.889829 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:24.890056 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:24.890140 (+ 84us) client_negotiation.cc:769] Sending connection context
0618 22:18:24.890229 (+ 89us) client_negotiation.cc:240] Negotiation successful
0618 22:18:24.890407 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"mutex_wait_us":230,"spinlock_wait_cycles":9728,"thread_start_us":78,"threads_started":1}
I20260618 22:18:24.890959 8522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:24.885196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:39557 (local address 127.0.98.66:39995)
0618 22:18:24.885455 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:24.885470 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:24.885642 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:24.886117 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:24.886120 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:24.886131 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:18:24.886215 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:24.886220 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.887290 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.887295 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:24.888853 (+ 1558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:24.888865 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.889809 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.889813 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:24.890060 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:24.890138 (+ 78us) client_negotiation.cc:769] Sending connection context
0618 22:18:24.890229 (+ 91us) client_negotiation.cc:240] Negotiation successful
0618 22:18:24.890398 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":248,"spinlock_wait_cycles":22656,"thread_start_us":67,"threads_started":1}
I20260618 22:18:24.891328 8370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48a3797d6c7444708666c32a3783f457" candidate_uuid: "e94305e8231b4fd9ad7f29754b581d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" is_pre_election: true
I20260618 22:18:24.891484 8370 raft_consensus.cc:2454] T 48a3797d6c7444708666c32a3783f457 P 2fb18f58d30d4e69998fd6b5bd106bff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e94305e8231b4fd9ad7f29754b581d84 in term 0.
I20260618 22:18:24.891435 8090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48a3797d6c7444708666c32a3783f457" candidate_uuid: "e94305e8231b4fd9ad7f29754b581d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" is_pre_election: true
I20260618 22:18:24.891602 8090 raft_consensus.cc:2454] T 48a3797d6c7444708666c32a3783f457 P f85c3f5cab7b4442b6f829cfbadd33b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e94305e8231b4fd9ad7f29754b581d84 in term 0.
I20260618 22:18:24.891829 8163 leader_election.cc:302] T 48a3797d6c7444708666c32a3783f457 P e94305e8231b4fd9ad7f29754b581d84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fb18f58d30d4e69998fd6b5bd106bff, e94305e8231b4fd9ad7f29754b581d84; no voters:
I20260618 22:18:24.892063 8451 raft_consensus.cc:2781] T 48a3797d6c7444708666c32a3783f457 P e94305e8231b4fd9ad7f29754b581d84 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:18:24.892118 8451 raft_consensus.cc:482] T 48a3797d6c7444708666c32a3783f457 P e94305e8231b4fd9ad7f29754b581d84 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:18:24.892155 8451 raft_consensus.cc:3037] T 48a3797d6c7444708666c32a3783f457 P e94305e8231b4fd9ad7f29754b581d84 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:18:24.893188 8451 raft_consensus.cc:504] T 48a3797d6c7444708666c32a3783f457 P e94305e8231b4fd9ad7f29754b581d84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 39557 } } peers { permanent_uuid: "e94305e8231b4fd9ad7f29754b581d84" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39465 } } peers { permanent_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39501 } }
I20260618 22:18:24.893517 8451 leader_election.cc:288] T 48a3797d6c7444708666c32a3783f457 P e94305e8231b4fd9ad7f29754b581d84 [CANDIDATE]: Term 1 election: Requested vote from peers f85c3f5cab7b4442b6f829cfbadd33b5 (127.0.98.65:39557), 2fb18f58d30d4e69998fd6b5bd106bff (127.0.98.67:39501)
I20260618 22:18:24.893849 8370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48a3797d6c7444708666c32a3783f457" candidate_uuid: "e94305e8231b4fd9ad7f29754b581d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fb18f58d30d4e69998fd6b5bd106bff"
I20260618 22:18:24.893872 8090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48a3797d6c7444708666c32a3783f457" candidate_uuid: "e94305e8231b4fd9ad7f29754b581d84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5"
I20260618 22:18:24.894003 8370 raft_consensus.cc:3037] T 48a3797d6c7444708666c32a3783f457 P 2fb18f58d30d4e69998fd6b5bd106bff [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:18:24.894032 8090 raft_consensus.cc:3037] T 48a3797d6c7444708666c32a3783f457 P f85c3f5cab7b4442b6f829cfbadd33b5 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:18:24.894872 8370 raft_consensus.cc:2454] T 48a3797d6c7444708666c32a3783f457 P 2fb18f58d30d4e69998fd6b5bd106bff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e94305e8231b4fd9ad7f29754b581d84 in term 1.
I20260618 22:18:24.895026 8090 raft_consensus.cc:2454] T 48a3797d6c7444708666c32a3783f457 P f85c3f5cab7b4442b6f829cfbadd33b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e94305e8231b4fd9ad7f29754b581d84 in term 1.
I20260618 22:18:24.895126 8163 leader_election.cc:302] T 48a3797d6c7444708666c32a3783f457 P e94305e8231b4fd9ad7f29754b581d84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fb18f58d30d4e69998fd6b5bd106bff, e94305e8231b4fd9ad7f29754b581d84; no voters:
I20260618 22:18:24.895414 8451 raft_consensus.cc:2781] T 48a3797d6c7444708666c32a3783f457 P e94305e8231b4fd9ad7f29754b581d84 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:18:24.895604 8451 raft_consensus.cc:686] T 48a3797d6c7444708666c32a3783f457 P e94305e8231b4fd9ad7f29754b581d84 [term 1 LEADER]: Becoming Leader. State: Replica: e94305e8231b4fd9ad7f29754b581d84, State: Running, Role: LEADER
I20260618 22:18:24.895849 8451 consensus_queue.cc:237] T 48a3797d6c7444708666c32a3783f457 P e94305e8231b4fd9ad7f29754b581d84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 39557 } } peers { permanent_uuid: "e94305e8231b4fd9ad7f29754b581d84" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39465 } } peers { permanent_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39501 } }
I20260618 22:18:24.898172 7933 catalog_manager.cc:5495] T 48a3797d6c7444708666c32a3783f457 P e94305e8231b4fd9ad7f29754b581d84 reported cstate change: term changed from 0 to 1, leader changed from <none> to e94305e8231b4fd9ad7f29754b581d84 (127.0.98.66). New cstate: current_term: 1 leader_uuid: "e94305e8231b4fd9ad7f29754b581d84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 39557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e94305e8231b4fd9ad7f29754b581d84" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39465 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39501 } health_report { overall_health: UNKNOWN } } }
I20260618 22:18:24.918488 8525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:24.914865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36908 (local address 127.0.98.66:39465)
0618 22:18:24.915099 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:24.915105 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:24.915116 (+ 11us) server_negotiation.cc:407] Connection header received
0618 22:18:24.915177 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:24.915179 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:24.915229 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:24.915327 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:18:24.915558 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.916143 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.916976 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.917255 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.917334 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:24.917560 (+ 226us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:18:24.918095 (+ 535us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:18:24.918207 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:24.918244 (+ 37us) server_negotiation.cc:299] Negotiation successful
0618 22:18:24.918294 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":99,"threads_started":1}
I20260618 22:18:24.928514 8528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:24.925244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:39501 (local address 127.0.98.66:50427)
0618 22:18:24.925546 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:24.925560 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:24.925652 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:24.925877 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:24.925879 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:24.925886 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0618 22:18:24.926064 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:24.926069 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.926804 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.926807 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:24.927422 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:24.927431 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.927897 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.927900 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:24.927977 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:24.928014 (+ 37us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0618 22:18:24.928282 (+ 268us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0618 22:18:24.928286 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 22:18:24.928337 (+ 51us) client_negotiation.cc:240] Negotiation successful
0618 22:18:24.928394 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":95,"threads_started":1}
I20260618 22:18:24.928507 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:24.925411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:50427 (local address 127.0.98.67:39501)
0618 22:18:24.925558 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:24.925561 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:24.925633 (+ 72us) server_negotiation.cc:407] Connection header received
0618 22:18:24.925739 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:24.925742 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:24.925785 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:24.925866 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:18:24.926162 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.926681 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.927537 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.927761 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.927848 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:24.928135 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:18:24.928199 (+ 64us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:18:24.928256 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:24.928364 (+ 108us) server_negotiation.cc:299] Negotiation successful
0618 22:18:24.928406 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260618 22:18:24.934345 8090 raft_consensus.cc:1261] T 48a3797d6c7444708666c32a3783f457 P f85c3f5cab7b4442b6f829cfbadd33b5 [term 1 FOLLOWER]: Refusing update from remote peer e94305e8231b4fd9ad7f29754b581d84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:18:24.934389 8370 raft_consensus.cc:1261] T 48a3797d6c7444708666c32a3783f457 P 2fb18f58d30d4e69998fd6b5bd106bff [term 1 FOLLOWER]: Refusing update from remote peer e94305e8231b4fd9ad7f29754b581d84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:18:24.934864 8451 consensus_queue.cc:1035] T 48a3797d6c7444708666c32a3783f457 P e94305e8231b4fd9ad7f29754b581d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fb18f58d30d4e69998fd6b5bd106bff" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 39501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:18:24.935315 8523 consensus_queue.cc:1035] T 48a3797d6c7444708666c32a3783f457 P e94305e8231b4fd9ad7f29754b581d84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f85c3f5cab7b4442b6f829cfbadd33b5" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 39557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:18:24.952596 8422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:24.948774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:39465 (local address 127.0.98.67:45247)
0618 22:18:24.948929 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:24.948946 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:24.949042 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:24.949431 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:24.949434 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:24.949441 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0618 22:18:24.949545 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:24.949550 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.950570 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.950574 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:24.951208 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:24.951216 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.951855 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.951860 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:24.951944 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:24.951976 (+ 32us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0618 22:18:24.952323 (+ 347us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0618 22:18:24.952329 (+ 6us) client_negotiation.cc:769] Sending connection context
0618 22:18:24.952397 (+ 68us) client_negotiation.cc:240] Negotiation successful
0618 22:18:24.952469 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260618 22:18:24.952618 8525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:24.948861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:45247 (local address 127.0.98.66:39465)
0618 22:18:24.948983 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:24.948988 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:24.949004 (+ 16us) server_negotiation.cc:407] Connection header received
0618 22:18:24.949268 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:24.949270 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:24.949313 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:24.949424 (+ 111us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:18:24.949664 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.950447 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.951344 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:24.951713 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:24.951826 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:24.952121 (+ 295us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:18:24.952226 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:18:24.952301 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:24.952423 (+ 122us) server_negotiation.cc:299] Negotiation successful
0618 22:18:24.952478 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260618 22:18:24.971903 7932 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:18:24.972064 7932 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38726:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:18:24.974058 7932 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ec613fe2b9494be494f0f98c8cfe0d63: Sending DeleteTablet for 3 replicas of tablet 48a3797d6c7444708666c32a3783f457
I20260618 22:18:24.974653 8070 tablet_service.cc:1514] Processing DeleteTablet for tablet 48a3797d6c7444708666c32a3783f457 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:18:24 UTC) from {username='oryx', principal='oryx/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:58350
I20260618 22:18:24.974831 8468 tablet_replica.cc:330] T 48a3797d6c7444708666c32a3783f457 P f85c3f5cab7b4442b6f829cfbadd33b5: stopping tablet replica
I20260618 22:18:24.974862 8209 tablet_service.cc:1514] Processing DeleteTablet for tablet 48a3797d6c7444708666c32a3783f457 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:18:24 UTC) from {username='oryx', principal='oryx/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:36894
I20260618 22:18:24.974961 8468 raft_consensus.cc:2229] T 48a3797d6c7444708666c32a3783f457 P f85c3f5cab7b4442b6f829cfbadd33b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:18:24.975028 8469 tablet_replica.cc:330] T 48a3797d6c7444708666c32a3783f457 P e94305e8231b4fd9ad7f29754b581d84: stopping tablet replica
I20260618 22:18:24.975063 8468 raft_consensus.cc:2258] T 48a3797d6c7444708666c32a3783f457 P f85c3f5cab7b4442b6f829cfbadd33b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:18:24.975135 8469 raft_consensus.cc:2229] T 48a3797d6c7444708666c32a3783f457 P e94305e8231b4fd9ad7f29754b581d84 [term 1 LEADER]: Raft consensus shutting down.
I20260618 22:18:24.975312 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 8005
I20260618 22:18:24.975400 8469 raft_consensus.cc:2258] T 48a3797d6c7444708666c32a3783f457 P e94305e8231b4fd9ad7f29754b581d84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:18:24.977758 8469 ts_tablet_manager.cc:1905] T 48a3797d6c7444708666c32a3783f457 P e94305e8231b4fd9ad7f29754b581d84: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:18:24.980266 8469 ts_tablet_manager.cc:1918] T 48a3797d6c7444708666c32a3783f457 P e94305e8231b4fd9ad7f29754b581d84: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260618 22:18:24.980361 8469 log.cc:1196] T 48a3797d6c7444708666c32a3783f457 P e94305e8231b4fd9ad7f29754b581d84: Deleting WAL directory at /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/ts-1/wal/wals/48a3797d6c7444708666c32a3783f457
I20260618 22:18:24.980605 8469 ts_tablet_manager.cc:1939] T 48a3797d6c7444708666c32a3783f457 P e94305e8231b4fd9ad7f29754b581d84: Deleting consensus metadata
I20260618 22:18:24.980825 8349 tablet_service.cc:1514] Processing DeleteTablet for tablet 48a3797d6c7444708666c32a3783f457 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:18:24 UTC) from {username='oryx', principal='oryx/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:45698
I20260618 22:18:24.980993 8470 tablet_replica.cc:330] T 48a3797d6c7444708666c32a3783f457 P 2fb18f58d30d4e69998fd6b5bd106bff: stopping tablet replica
I20260618 22:18:24.981091 8470 raft_consensus.cc:2229] T 48a3797d6c7444708666c32a3783f457 P 2fb18f58d30d4e69998fd6b5bd106bff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:18:24.981202 8470 raft_consensus.cc:2258] T 48a3797d6c7444708666c32a3783f457 P 2fb18f58d30d4e69998fd6b5bd106bff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:18:24.981452 7919 catalog_manager.cc:4841] TS e94305e8231b4fd9ad7f29754b581d84 (127.0.98.66:39465): tablet 48a3797d6c7444708666c32a3783f457 (table test-table [id=35a7bdaea358412096e98521aed2c102]) successfully deleted
I20260618 22:18:24.981967 8470 ts_tablet_manager.cc:1905] T 48a3797d6c7444708666c32a3783f457 P 2fb18f58d30d4e69998fd6b5bd106bff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:18:24.983943 8470 ts_tablet_manager.cc:1918] T 48a3797d6c7444708666c32a3783f457 P 2fb18f58d30d4e69998fd6b5bd106bff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260618 22:18:24.984046 8470 log.cc:1196] T 48a3797d6c7444708666c32a3783f457 P 2fb18f58d30d4e69998fd6b5bd106bff: Deleting WAL directory at /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781821065380026-393-0/minicluster-data/ts-2/wal/wals/48a3797d6c7444708666c32a3783f457
I20260618 22:18:24.984292 8470 ts_tablet_manager.cc:1939] T 48a3797d6c7444708666c32a3783f457 P 2fb18f58d30d4e69998fd6b5bd106bff: Deleting consensus metadata
I20260618 22:18:24.984928 7919 catalog_manager.cc:4841] TS 2fb18f58d30d4e69998fd6b5bd106bff (127.0.98.67:39501): tablet 48a3797d6c7444708666c32a3783f457 (table test-table [id=35a7bdaea358412096e98521aed2c102]) successfully deleted
W20260618 22:18:24.988168 7918 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.65:39557 (error 108)
I20260618 22:18:24.988431 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 8145
I20260618 22:18:24.990170 8534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:24.989007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:39557 (local address 127.0.0.1:58376)
0618 22:18:24.989964 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:24.990044 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.65:39557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":856,"thread_start_us":61,"threads_started":1}
W20260618 22:18:24.990291 7918 catalog_manager.cc:4584] TS f85c3f5cab7b4442b6f829cfbadd33b5 (127.0.98.65:39557): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 48a3797d6c7444708666c32a3783f457: Network error: Client connection negotiation failed: client connection to 127.0.98.65:39557: connect: Connection refused (error 111)
I20260618 22:18:24.995154 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 8285
I20260618 22:18:25.002795 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 7901
2026-06-18T22:18:25Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3011 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781821065380026-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:18:25 dist-test-slave-g6r4 krb5kdc[8537](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:18:25 dist-test-slave-g6r4 krb5kdc[8537](info): set up 2 sockets
Jun 18 22:18:25 dist-test-slave-g6r4 krb5kdc[8537](info): commencing operation
krb5kdc: starting...
W20260618 22:18:26.046559 393 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781821065380026-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781821065380026-393-0/krb5kdc/test-user.keytab.
Jun 18 22:18:26 dist-test-slave-g6r4 krb5kdc[8537](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821106, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:18:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:18:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260618 22:18:26.164343 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781821065380026-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781821065380026-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781821065380026-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:39491
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:45199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:39491
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:18:26.270493 8550 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:18:26.270722 8550 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:18:26.270759 8550 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:18:26.274602 8550 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:18:26.274646 8550 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:18:26.274664 8550 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:18:26.274680 8550 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:18:26.274695 8550 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:18:26.279629 8550 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:45199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781821065380026-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781821065380026-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:39491
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:39491
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8550
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781821065380026-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:18:26.280505 8550 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:18:26.281399 8550 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:18:26.289703 8555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:26.289812 8558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:26.289749 8556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:26.290019 8550 server_base.cc:1034] running on GCE node
I20260618 22:18:26.290589 8550 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:18:26.291620 8550 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:18:26.292812 8550 hybrid_clock.cc:648] HybridClock initialized: now 1781821106292798 us; error 35 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-18T22:18:26Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1297 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:18:26 dist-test-slave-g6r4 krb5kdc[8563](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:18:26 dist-test-slave-g6r4 krb5kdc[8563](info): set up 2 sockets
Jun 18 22:18:26 dist-test-slave-g6r4 krb5kdc[8563](info): commencing operation
krb5kdc: starting...
W20260618 22:18:27.338678 393 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/krb5kdc/test-user.keytab.
Jun 18 22:18:27 dist-test-slave-g6r4 krb5kdc[8563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:18:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:18:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260618 22:18:27.456167 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:37613
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:36987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:37613
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:18:27.563040 8576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:18:27.563275 8576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:18:27.563314 8576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:18:27.567202 8576 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:18:27.567257 8576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:18:27.567276 8576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:18:27.567293 8576 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:18:27.567308 8576 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:18:27.572377 8576 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:36987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:37613
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:37613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:18:27.573256 8576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:18:27.574216 8576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:18:27.582370 8581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:27.582376 8582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:27.582422 8576 server_base.cc:1034] running on GCE node
W20260618 22:18:27.582377 8584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:27.583170 8576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:18:27.584522 8576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:18:27.585785 8576 hybrid_clock.cc:648] HybridClock initialized: now 1781821107585748 us; error 53 us; skew 500 ppm
Jun 18 22:18:27 dist-test-slave-g6r4 krb5kdc[8563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821107, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:18:27.592594 8576 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260618 22:18:27.593434 8576 webserver.cc:460] Webserver started at http://127.0.98.126:43625/ using document root <none> and password file <none>
I20260618 22:18:27.594031 8576 fs_manager.cc:359] Metadata directory not provided
I20260618 22:18:27.594118 8576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:18:27.594317 8576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:18:27.596246 8576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/master-0/data/instance:
uuid: "eadb8a55f4cb436da6a5292accde36e8"
format_stamp: "Formatted at 2026-06-18 22:18:27 on dist-test-slave-g6r4"
server_key: "2daf14bcc44150f19346039dcad5b7cd"
server_key_iv: "1029301f92e527e22ec4f1a9f2b7ec00"
server_key_version: "encryptionkey@0"
I20260618 22:18:27.596804 8576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/master-0/wal/instance:
uuid: "eadb8a55f4cb436da6a5292accde36e8"
format_stamp: "Formatted at 2026-06-18 22:18:27 on dist-test-slave-g6r4"
server_key: "2daf14bcc44150f19346039dcad5b7cd"
server_key_iv: "1029301f92e527e22ec4f1a9f2b7ec00"
server_key_version: "encryptionkey@0"
I20260618 22:18:27.600286 8576 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260618 22:18:27.603188 8591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:27.604413 8576 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260618 22:18:27.604557 8576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/master-0/wal
uuid: "eadb8a55f4cb436da6a5292accde36e8"
format_stamp: "Formatted at 2026-06-18 22:18:27 on dist-test-slave-g6r4"
server_key: "2daf14bcc44150f19346039dcad5b7cd"
server_key_iv: "1029301f92e527e22ec4f1a9f2b7ec00"
server_key_version: "encryptionkey@0"
I20260618 22:18:27.604700 8576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:18:27.631762 8576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:18:27.635223 8576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:18:27.635434 8576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:18:27.642252 8576 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:37613
I20260618 22:18:27.642274 8643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:37613 every 8 connection(s)
I20260618 22:18:27.643184 8576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
I20260618 22:18:27.645835 8644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:18:27.651072 8644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eadb8a55f4cb436da6a5292accde36e8: Bootstrap starting.
I20260618 22:18:27.652093 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 8576
I20260618 22:18:27.652212 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/master-0/wal/instance
I20260618 22:18:27.652518 393 external_mini_cluster.cc:1428] Setting key 07853e96ee6b7adbb96c29b7e0ff9de7
I20260618 22:18:27.653744 8644 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eadb8a55f4cb436da6a5292accde36e8: Neither blocks nor log segments found. Creating new log.
I20260618 22:18:27.654651 8644 log.cc:824] T 00000000000000000000000000000000 P eadb8a55f4cb436da6a5292accde36e8: Log is configured to *not* fsync() on all Append() calls
I20260618 22:18:27.656757 8644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eadb8a55f4cb436da6a5292accde36e8: No bootstrap required, opened a new log
I20260618 22:18:27.661552 8644 raft_consensus.cc:348] T 00000000000000000000000000000000 P eadb8a55f4cb436da6a5292accde36e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eadb8a55f4cb436da6a5292accde36e8" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 37613 } }
I20260618 22:18:27.661887 8644 raft_consensus.cc:374] T 00000000000000000000000000000000 P eadb8a55f4cb436da6a5292accde36e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:18:27.662010 8644 raft_consensus.cc:729] T 00000000000000000000000000000000 P eadb8a55f4cb436da6a5292accde36e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eadb8a55f4cb436da6a5292accde36e8, State: Initialized, Role: FOLLOWER
Jun 18 22:18:27 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821107, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:18:27.662570 8644 consensus_queue.cc:260] T 00000000000000000000000000000000 P eadb8a55f4cb436da6a5292accde36e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eadb8a55f4cb436da6a5292accde36e8" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 37613 } }
I20260618 22:18:27.662746 8644 raft_consensus.cc:388] T 00000000000000000000000000000000 P eadb8a55f4cb436da6a5292accde36e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:18:27.662882 8644 raft_consensus.cc:482] T 00000000000000000000000000000000 P eadb8a55f4cb436da6a5292accde36e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:18:27.662997 8644 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eadb8a55f4cb436da6a5292accde36e8 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:18:27.664199 8644 raft_consensus.cc:504] T 00000000000000000000000000000000 P eadb8a55f4cb436da6a5292accde36e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eadb8a55f4cb436da6a5292accde36e8" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 37613 } }
I20260618 22:18:27.664566 8644 leader_election.cc:302] T 00000000000000000000000000000000 P eadb8a55f4cb436da6a5292accde36e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eadb8a55f4cb436da6a5292accde36e8; no voters:
I20260618 22:18:27.664929 8644 leader_election.cc:288] T 00000000000000000000000000000000 P eadb8a55f4cb436da6a5292accde36e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:18:27.665035 8649 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eadb8a55f4cb436da6a5292accde36e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:18:27.665282 8649 raft_consensus.cc:686] T 00000000000000000000000000000000 P eadb8a55f4cb436da6a5292accde36e8 [term 1 LEADER]: Becoming Leader. State: Replica: eadb8a55f4cb436da6a5292accde36e8, State: Running, Role: LEADER
I20260618 22:18:27.666101 8649 consensus_queue.cc:237] T 00000000000000000000000000000000 P eadb8a55f4cb436da6a5292accde36e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eadb8a55f4cb436da6a5292accde36e8" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 37613 } }
I20260618 22:18:27.666323 8644 sys_catalog.cc:564] T 00000000000000000000000000000000 P eadb8a55f4cb436da6a5292accde36e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:18:27.667743 8651 sys_catalog.cc:455] T 00000000000000000000000000000000 P eadb8a55f4cb436da6a5292accde36e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eadb8a55f4cb436da6a5292accde36e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eadb8a55f4cb436da6a5292accde36e8" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 37613 } } }
I20260618 22:18:27.667861 8651 sys_catalog.cc:458] T 00000000000000000000000000000000 P eadb8a55f4cb436da6a5292accde36e8 [sys.catalog]: This master's current role is: LEADER
I20260618 22:18:27.668242 8658 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:18:27.668273 8650 sys_catalog.cc:455] T 00000000000000000000000000000000 P eadb8a55f4cb436da6a5292accde36e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eadb8a55f4cb436da6a5292accde36e8. Latest consensus state: current_term: 1 leader_uuid: "eadb8a55f4cb436da6a5292accde36e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eadb8a55f4cb436da6a5292accde36e8" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 37613 } } }
I20260618 22:18:27.668416 8650 sys_catalog.cc:458] T 00000000000000000000000000000000 P eadb8a55f4cb436da6a5292accde36e8 [sys.catalog]: This master's current role is: LEADER
I20260618 22:18:27.669323 8647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:27.653611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33640 (local address 127.0.98.126:37613)
0618 22:18:27.654095 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:27.654114 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:27.654155 (+ 41us) server_negotiation.cc:407] Connection header received
0618 22:18:27.654995 (+ 840us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:27.655020 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:27.655409 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:27.655755 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:27.656417 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:27.657832 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:27.659114 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:27.659885 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:27.660076 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:27.663351 (+ 3275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:27.663359 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:27.663377 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:27.663402 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:27.665414 (+ 2012us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:27.665891 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:27.665927 (+ 36us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:27.665934 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:27.666069 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:27.666398 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:27.666401 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:27.666403 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:27.666903 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:27.667783 (+ 880us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:27.667940 (+ 157us) server_negotiation.cc:299] Negotiation successful
0618 22:18:27.668231 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":138,"threads_started":1}
I20260618 22:18:27.672209 8658 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:18:27.677450 8658 catalog_manager.cc:1347] Generated new cluster ID: b76aac40ea9340a795f096cf3aab0f7b
I20260618 22:18:27.677505 8658 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:18:27.683995 8658 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:18:27.684878 8658 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:18:27.690016 8658 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eadb8a55f4cb436da6a5292accde36e8: Generated new TSK 0
I20260618 22:18:27.690596 8658 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab.
I20260618 22:18:27.723094 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:37613
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:36987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:18:27.852869 8670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:18:27.853137 8670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:18:27.853215 8670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:18:27.857272 8670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:18:27.857352 8670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:18:27.857441 8670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260618 22:18:27.862552 8670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:36987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:37613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:18:27.863500 8670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:18:27.864455 8670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:18:27.873602 8678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:27.873661 8676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:27.873600 8675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:27.873983 8670 server_base.cc:1034] running on GCE node
I20260618 22:18:27.874351 8670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:18:27.874979 8670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:18:27.876163 8670 hybrid_clock.cc:648] HybridClock initialized: now 1781821107876141 us; error 37 us; skew 500 ppm
Jun 18 22:18:27 dist-test-slave-g6r4 krb5kdc[8563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821107, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:18:27.878851 8670 init.cc:377] Logged in from keytab as kudu/127.0.98.65@KRBTEST.COM (short username kudu)
I20260618 22:18:27.879680 8670 webserver.cc:460] Webserver started at http://127.0.98.65:44569/ using document root <none> and password file <none>
I20260618 22:18:27.880209 8670 fs_manager.cc:359] Metadata directory not provided
I20260618 22:18:27.880298 8670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:18:27.880501 8670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:18:27.882473 8670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-0/data/instance:
uuid: "4c4e50c0891340a4a8b0f7bd425ea8f2"
format_stamp: "Formatted at 2026-06-18 22:18:27 on dist-test-slave-g6r4"
server_key: "d4b143f4663b9ab3ca774b3d8630d88f"
server_key_iv: "e6c54af835833f4e1075f2753995c537"
server_key_version: "encryptionkey@0"
I20260618 22:18:27.883028 8670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-0/wal/instance:
uuid: "4c4e50c0891340a4a8b0f7bd425ea8f2"
format_stamp: "Formatted at 2026-06-18 22:18:27 on dist-test-slave-g6r4"
server_key: "d4b143f4663b9ab3ca774b3d8630d88f"
server_key_iv: "e6c54af835833f4e1075f2753995c537"
server_key_version: "encryptionkey@0"
I20260618 22:18:27.886515 8670 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260618 22:18:27.888839 8685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:27.890201 8670 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260618 22:18:27.890290 8670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-0/wal
uuid: "4c4e50c0891340a4a8b0f7bd425ea8f2"
format_stamp: "Formatted at 2026-06-18 22:18:27 on dist-test-slave-g6r4"
server_key: "d4b143f4663b9ab3ca774b3d8630d88f"
server_key_iv: "e6c54af835833f4e1075f2753995c537"
server_key_version: "encryptionkey@0"
I20260618 22:18:27.890414 8670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:18:27.903774 8670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:18:27.906580 8670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:18:27.906765 8670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:18:27.907310 8670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:18:27.908108 8670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:18:27.908164 8670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:27.908226 8670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:18:27.908257 8670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:27.917018 8670 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:46105
I20260618 22:18:27.917045 8798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:46105 every 8 connection(s)
I20260618 22:18:27.917822 8670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
I20260618 22:18:27.919505 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 8670
I20260618 22:18:27.919628 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-0/wal/instance
I20260618 22:18:27.919921 393 external_mini_cluster.cc:1428] Setting key fe9b69de4c11b099e05d6117ac1af2a5
Jun 18 22:18:27 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821107, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.66@KRBTEST.COM" created.
I20260618 22:18:27.933697 8647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:27.919613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:41355 (local address 127.0.98.126:37613)
0618 22:18:27.919746 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:27.919751 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:27.920789 (+ 1038us) server_negotiation.cc:407] Connection header received
0618 22:18:27.922493 (+ 1704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:27.922497 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:27.922548 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:27.922674 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:27.924333 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:27.924855 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:27.926902 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:27.927136 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:27.927277 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:27.930089 (+ 2812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:27.930092 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:27.930094 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:27.930116 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:27.931572 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:27.932147 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:27.932152 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:27.932154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:27.932229 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:27.932611 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:27.932615 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:27.932616 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:27.932823 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:27.932925 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:27.933408 (+ 483us) server_negotiation.cc:299] Negotiation successful
0618 22:18:27.933525 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260618 22:18:27.934482 8801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:27.920070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37613 (local address 127.0.98.65:41355)
0618 22:18:27.920619 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:27.920662 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:27.922283 (+ 1621us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:27.922867 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:27.922879 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:27.923438 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:27.924155 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:27.924178 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:27.924992 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:27.924999 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:27.926768 (+ 1769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:27.926783 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:27.927321 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:27.927327 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:27.927581 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:27.928195 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:27.928227 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:27.929867 (+ 1640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:18:27.931699 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:27.931704 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:27.931718 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:27.932037 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:27.932323 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:27.932326 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:27.932328 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:27.932504 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:27.932935 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:18:27.932943 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:18:27.933168 (+ 225us) client_negotiation.cc:769] Sending connection context
0618 22:18:27.933346 (+ 178us) client_negotiation.cc:240] Negotiation successful
0618 22:18:27.933647 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":141,"threads_started":1}
I20260618 22:18:27.935724 8799 heartbeater.cc:347] Connected to a master server at 127.0.98.126:37613
I20260618 22:18:27.935981 8799 heartbeater.cc:464] Registering TS with master...
I20260618 22:18:27.936602 8799 heartbeater.cc:511] Master 127.0.98.126:37613 requested a full tablet report, sending...
I20260618 22:18:27.938431 8608 ts_manager.cc:194] Registered new tserver with Master: 4c4e50c0891340a4a8b0f7bd425ea8f2 (127.0.98.65:46105)
I20260618 22:18:27.940253 8608 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.65@KRBTEST.COM'} at 127.0.98.65:41355
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab.
I20260618 22:18:27.949115 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:37613
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:36987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:18:28.066506 8804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:18:28.066753 8804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:18:28.066824 8804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:18:28.071724 8804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:18:28.071786 8804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:18:28.071909 8804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260618 22:18:28.077075 8804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:36987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:37613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:18:28.078079 8804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:18:28.079030 8804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:18:28.088251 8809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:28.088292 8812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:28.088343 8810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:28.088580 8804 server_base.cc:1034] running on GCE node
I20260618 22:18:28.088953 8804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:18:28.089569 8804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:18:28.090739 8804 hybrid_clock.cc:648] HybridClock initialized: now 1781821108090706 us; error 43 us; skew 500 ppm
Jun 18 22:18:28 dist-test-slave-g6r4 krb5kdc[8563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821108, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:18:28.093380 8804 init.cc:377] Logged in from keytab as kudu/127.0.98.66@KRBTEST.COM (short username kudu)
I20260618 22:18:28.094303 8804 webserver.cc:460] Webserver started at http://127.0.98.66:46707/ using document root <none> and password file <none>
I20260618 22:18:28.094853 8804 fs_manager.cc:359] Metadata directory not provided
I20260618 22:18:28.094913 8804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:18:28.095065 8804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:18:28.096897 8804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-1/data/instance:
uuid: "fbc3e88ddcc1435ab0ebf3cf49928862"
format_stamp: "Formatted at 2026-06-18 22:18:28 on dist-test-slave-g6r4"
server_key: "0a64f9b738bf30a6fae05951cb09bf6c"
server_key_iv: "6038ef8fe169cb4e0381b340424123fd"
server_key_version: "encryptionkey@0"
I20260618 22:18:28.097380 8804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-1/wal/instance:
uuid: "fbc3e88ddcc1435ab0ebf3cf49928862"
format_stamp: "Formatted at 2026-06-18 22:18:28 on dist-test-slave-g6r4"
server_key: "0a64f9b738bf30a6fae05951cb09bf6c"
server_key_iv: "6038ef8fe169cb4e0381b340424123fd"
server_key_version: "encryptionkey@0"
I20260618 22:18:28.100911 8804 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260618 22:18:28.103188 8819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:28.104383 8804 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260618 22:18:28.104471 8804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-1/wal
uuid: "fbc3e88ddcc1435ab0ebf3cf49928862"
format_stamp: "Formatted at 2026-06-18 22:18:28 on dist-test-slave-g6r4"
server_key: "0a64f9b738bf30a6fae05951cb09bf6c"
server_key_iv: "6038ef8fe169cb4e0381b340424123fd"
server_key_version: "encryptionkey@0"
I20260618 22:18:28.104547 8804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:18:28.131067 8804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:18:28.133844 8804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:18:28.134016 8804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:18:28.134508 8804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:18:28.135279 8804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:18:28.135326 8804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:28.135372 8804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:18:28.135396 8804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:28.144621 8804 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:34785
I20260618 22:18:28.144737 8932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:34785 every 8 connection(s)
I20260618 22:18:28.145553 8804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
Jun 18 22:18:28 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821108, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:18:28.155550 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 8804
I20260618 22:18:28.155782 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-1/wal/instance
I20260618 22:18:28.156138 393 external_mini_cluster.cc:1428] Setting key 204ed39d12951a8cd0ca737be1239546
I20260618 22:18:28.159770 8647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:28.147361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:50129 (local address 127.0.98.126:37613)
0618 22:18:28.147517 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:28.147520 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:28.148322 (+ 802us) server_negotiation.cc:407] Connection header received
0618 22:18:28.149916 (+ 1594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:28.149920 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:28.149979 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:28.150103 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:28.151274 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:28.151844 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:28.153071 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:28.153321 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:28.153442 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:28.155947 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:28.155953 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:28.155956 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:28.155989 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:28.157634 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:28.158246 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:28.158250 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:28.158256 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:28.158351 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:28.158665 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:28.158671 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:28.158676 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:28.158936 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:28.159047 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:28.159487 (+ 440us) server_negotiation.cc:299] Negotiation successful
0618 22:18:28.159605 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260618 22:18:28.160532 8935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:28.147708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37613 (local address 127.0.98.66:50129)
0618 22:18:28.148157 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:28.148204 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:28.149636 (+ 1432us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:28.150281 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:28.150288 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:28.150647 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:28.151119 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:28.151132 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:28.152000 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:28.152004 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:28.152935 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:28.152944 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:28.153472 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:28.153476 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:28.153655 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:28.154314 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:28.154331 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:28.155776 (+ 1445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:18:28.157768 (+ 1992us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:28.157772 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:28.157782 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:28.158142 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:28.158447 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:28.158449 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:28.158450 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:28.158573 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:28.159039 (+ 466us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:18:28.159044 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:18:28.159252 (+ 208us) client_negotiation.cc:769] Sending connection context
0618 22:18:28.159445 (+ 193us) client_negotiation.cc:240] Negotiation successful
0618 22:18:28.159741 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":115,"threads_started":1}
I20260618 22:18:28.161619 8933 heartbeater.cc:347] Connected to a master server at 127.0.98.126:37613
I20260618 22:18:28.161864 8933 heartbeater.cc:464] Registering TS with master...
I20260618 22:18:28.162379 8933 heartbeater.cc:511] Master 127.0.98.126:37613 requested a full tablet report, sending...
I20260618 22:18:28.163447 8608 ts_manager.cc:194] Registered new tserver with Master: fbc3e88ddcc1435ab0ebf3cf49928862 (127.0.98.66:34785)
I20260618 22:18:28.164389 8608 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.66@KRBTEST.COM'} at 127.0.98.66:50129
WARNING: no policy specified for kudu/127.0.98.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab.
I20260618 22:18:28.183145 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:37613
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:36987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:18:28.289193 8938 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:18:28.289439 8938 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:18:28.289520 8938 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:18:28.293495 8938 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:18:28.293581 8938 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:18:28.293677 8938 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260618 22:18:28.298825 8938 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:36987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:37613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8938
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:18:28.299777 8938 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:18:28.300741 8938 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:18:28.310094 8943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:28.310158 8938 server_base.cc:1034] running on GCE node
W20260618 22:18:28.310096 8946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:28.310339 8944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:28.310729 8938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:18:28.311375 8938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:18:28.312534 8938 hybrid_clock.cc:648] HybridClock initialized: now 1781821108312517 us; error 37 us; skew 500 ppm
Jun 18 22:18:28 dist-test-slave-g6r4 krb5kdc[8563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821108, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:18:28.315209 8938 init.cc:377] Logged in from keytab as kudu/127.0.98.67@KRBTEST.COM (short username kudu)
I20260618 22:18:28.315984 8938 webserver.cc:460] Webserver started at http://127.0.98.67:43495/ using document root <none> and password file <none>
I20260618 22:18:28.316469 8938 fs_manager.cc:359] Metadata directory not provided
I20260618 22:18:28.316525 8938 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:18:28.316677 8938 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:18:28.318668 8938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-2/data/instance:
uuid: "81e11ef0cf8d49e4bcd65052106f9cdd"
format_stamp: "Formatted at 2026-06-18 22:18:28 on dist-test-slave-g6r4"
server_key: "197485229128572163be4942ba6db711"
server_key_iv: "f99e9705b28ba8cd7e7c8f200f54796a"
server_key_version: "encryptionkey@0"
I20260618 22:18:28.319154 8938 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-2/wal/instance:
uuid: "81e11ef0cf8d49e4bcd65052106f9cdd"
format_stamp: "Formatted at 2026-06-18 22:18:28 on dist-test-slave-g6r4"
server_key: "197485229128572163be4942ba6db711"
server_key_iv: "f99e9705b28ba8cd7e7c8f200f54796a"
server_key_version: "encryptionkey@0"
I20260618 22:18:28.322714 8938 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:18:28.325506 8953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:28.326808 8938 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:18:28.326960 8938 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-2/wal
uuid: "81e11ef0cf8d49e4bcd65052106f9cdd"
format_stamp: "Formatted at 2026-06-18 22:18:28 on dist-test-slave-g6r4"
server_key: "197485229128572163be4942ba6db711"
server_key_iv: "f99e9705b28ba8cd7e7c8f200f54796a"
server_key_version: "encryptionkey@0"
I20260618 22:18:28.327069 8938 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:18:28.343297 8938 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:18:28.346240 8938 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:18:28.346431 8938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:18:28.346982 8938 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:18:28.347747 8938 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:18:28.347816 8938 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:28.347909 8938 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:18:28.347959 8938 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:28.356707 8938 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:41329
I20260618 22:18:28.356734 9066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:41329 every 8 connection(s)
I20260618 22:18:28.357529 8938 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
I20260618 22:18:28.358678 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 8938
I20260618 22:18:28.358824 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-2/wal/instance
I20260618 22:18:28.359097 393 external_mini_cluster.cc:1428] Setting key 335eaf08bb027d0b4994636890479d3b
Jun 18 22:18:28 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821108, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:18:28.370638 8647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:28.359264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:37081 (local address 127.0.98.126:37613)
0618 22:18:28.359425 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:28.359437 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:28.360327 (+ 890us) server_negotiation.cc:407] Connection header received
0618 22:18:28.361662 (+ 1335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:28.361664 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:28.361706 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:28.361772 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:28.362808 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:28.363310 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:28.364445 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:28.364644 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:28.364716 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:28.367268 (+ 2552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:28.367278 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:28.367281 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:28.367302 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:28.368778 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:28.369278 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:28.369280 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:28.369281 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:28.369360 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:28.369659 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:28.369661 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:28.369662 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:28.369814 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:28.369888 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:28.370421 (+ 533us) server_negotiation.cc:299] Negotiation successful
0618 22:18:28.370509 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260618 22:18:28.371469 9069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:28.359729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37613 (local address 127.0.98.67:37081)
0618 22:18:28.360167 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:28.360216 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:28.361477 (+ 1261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:28.361917 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:28.361923 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:28.362251 (+ 328us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:28.362685 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:28.362697 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:28.363430 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:28.363433 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:28.364345 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:28.364354 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:28.364791 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:28.364793 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:28.364921 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:28.365507 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:28.365522 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:28.367121 (+ 1599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:18:28.368886 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:28.368891 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:28.368901 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:28.369173 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:28.369445 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:28.369447 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:28.369449 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:28.369580 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:28.369959 (+ 379us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:18:28.369964 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:18:28.370187 (+ 223us) client_negotiation.cc:769] Sending connection context
0618 22:18:28.370372 (+ 185us) client_negotiation.cc:240] Negotiation successful
0618 22:18:28.370627 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":134,"threads_started":1}
I20260618 22:18:28.372577 9067 heartbeater.cc:347] Connected to a master server at 127.0.98.126:37613
I20260618 22:18:28.372828 9067 heartbeater.cc:464] Registering TS with master...
I20260618 22:18:28.373294 9067 heartbeater.cc:511] Master 127.0.98.126:37613 requested a full tablet report, sending...
I20260618 22:18:28.374286 8608 ts_manager.cc:194] Registered new tserver with Master: 81e11ef0cf8d49e4bcd65052106f9cdd (127.0.98.67:41329)
I20260618 22:18:28.374780 8608 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.67@KRBTEST.COM'} at 127.0.98.67:37081
I20260618 22:18:28.384415 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/krb5kdc/oryx_127.0.98.65.keytab.
Entry for principal oryx/127.0.98.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/krb5kdc/oryx_127.0.98.65.keytab.
WARNING: no policy specified for oryx/127.0.98.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/krb5kdc/oryx_127.0.98.66.keytab.
Entry for principal oryx/127.0.98.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/krb5kdc/oryx_127.0.98.66.keytab.
WARNING: no policy specified for oryx/127.0.98.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/krb5kdc/oryx_127.0.98.67.keytab.
Entry for principal oryx/127.0.98.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/krb5kdc/oryx_127.0.98.67.keytab.
I20260618 22:18:28.454324 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 8670
I20260618 22:18:28.459587 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 8804
I20260618 22:18:28.464726 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 8938
I20260618 22:18:28.470297 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 8576
I20260618 22:18:28.476550 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:37613
--webserver_interface=127.0.98.126
--webserver_port=43625
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:36987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:37613
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:18:28.594251 9076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:18:28.594584 9076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:18:28.594661 9076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:18:28.602562 9076 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:18:28.602653 9076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:18:28.602694 9076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:18:28.602742 9076 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:18:28.602782 9076 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:18:28.613685 9076 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:36987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:37613
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:37613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=43625
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9076
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:18:28.615222 9076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:18:28.616194 9076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:18:28.624692 9081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:28.624781 9076 server_base.cc:1034] running on GCE node
W20260618 22:18:28.624706 9084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:28.624655 9082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:28.625401 9076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:18:28.626529 9076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:18:28.627714 9076 hybrid_clock.cc:648] HybridClock initialized: now 1781821108627698 us; error 28 us; skew 500 ppm
Jun 18 22:18:28 dist-test-slave-g6r4 krb5kdc[8563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821108, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:18:28.630607 9076 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260618 22:18:28.631513 9076 webserver.cc:460] Webserver started at http://127.0.98.126:43625/ using document root <none> and password file <none>
I20260618 22:18:28.632072 9076 fs_manager.cc:359] Metadata directory not provided
I20260618 22:18:28.632176 9076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:18:28.635980 9076 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260618 22:18:28.638139 9091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:28.639443 9076 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:18:28.639582 9076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/master-0/wal
uuid: "eadb8a55f4cb436da6a5292accde36e8"
format_stamp: "Formatted at 2026-06-18 22:18:27 on dist-test-slave-g6r4"
server_key: "2daf14bcc44150f19346039dcad5b7cd"
server_key_iv: "1029301f92e527e22ec4f1a9f2b7ec00"
server_key_version: "encryptionkey@0"
I20260618 22:18:28.640107 9076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:18:28.663640 9076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:18:28.666940 9076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:18:28.667143 9076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:18:28.674317 9076 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:37613
I20260618 22:18:28.674345 9143 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:37613 every 8 connection(s)
I20260618 22:18:28.675313 9076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
I20260618 22:18:28.679278 9144 sys_catalog.cc:263] Verifying existing consensus state
I20260618 22:18:28.680830 9144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eadb8a55f4cb436da6a5292accde36e8: Bootstrap starting.
I20260618 22:18:28.682976 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 9076
I20260618 22:18:28.683396 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:46105
--local_ip_for_outbound_sockets=127.0.98.65
--tserver_master_addrs=127.0.98.126:37613
--webserver_port=44569
--webserver_interface=127.0.98.65
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:36987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.65
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/krb5kdc/oryx_127.0.98.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 22:18:28.702145 9144 log.cc:824] T 00000000000000000000000000000000 P eadb8a55f4cb436da6a5292accde36e8: Log is configured to *not* fsync() on all Append() calls
I20260618 22:18:28.710551 9144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eadb8a55f4cb436da6a5292accde36e8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260618 22:18:28.710970 9144 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eadb8a55f4cb436da6a5292accde36e8: Bootstrap complete.
I20260618 22:18:28.716846 9144 raft_consensus.cc:348] T 00000000000000000000000000000000 P eadb8a55f4cb436da6a5292accde36e8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eadb8a55f4cb436da6a5292accde36e8" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 37613 } }
I20260618 22:18:28.717377 9144 raft_consensus.cc:729] T 00000000000000000000000000000000 P eadb8a55f4cb436da6a5292accde36e8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: eadb8a55f4cb436da6a5292accde36e8, State: Initialized, Role: FOLLOWER
I20260618 22:18:28.717971 9144 consensus_queue.cc:260] T 00000000000000000000000000000000 P eadb8a55f4cb436da6a5292accde36e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eadb8a55f4cb436da6a5292accde36e8" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 37613 } }
I20260618 22:18:28.718127 9144 raft_consensus.cc:388] T 00000000000000000000000000000000 P eadb8a55f4cb436da6a5292accde36e8 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:18:28.718209 9144 raft_consensus.cc:482] T 00000000000000000000000000000000 P eadb8a55f4cb436da6a5292accde36e8 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:18:28.718304 9144 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eadb8a55f4cb436da6a5292accde36e8 [term 1 FOLLOWER]: Advancing to term 2
I20260618 22:18:28.720847 9144 raft_consensus.cc:504] T 00000000000000000000000000000000 P eadb8a55f4cb436da6a5292accde36e8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eadb8a55f4cb436da6a5292accde36e8" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 37613 } }
I20260618 22:18:28.721218 9144 leader_election.cc:302] T 00000000000000000000000000000000 P eadb8a55f4cb436da6a5292accde36e8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eadb8a55f4cb436da6a5292accde36e8; no voters:
I20260618 22:18:28.721575 9144 leader_election.cc:288] T 00000000000000000000000000000000 P eadb8a55f4cb436da6a5292accde36e8 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260618 22:18:28.721706 9148 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eadb8a55f4cb436da6a5292accde36e8 [term 2 FOLLOWER]: Leader election won for term 2
I20260618 22:18:28.721989 9148 raft_consensus.cc:686] T 00000000000000000000000000000000 P eadb8a55f4cb436da6a5292accde36e8 [term 2 LEADER]: Becoming Leader. State: Replica: eadb8a55f4cb436da6a5292accde36e8, State: Running, Role: LEADER
I20260618 22:18:28.722328 9148 consensus_queue.cc:237] T 00000000000000000000000000000000 P eadb8a55f4cb436da6a5292accde36e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eadb8a55f4cb436da6a5292accde36e8" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 37613 } }
I20260618 22:18:28.722875 9144 sys_catalog.cc:564] T 00000000000000000000000000000000 P eadb8a55f4cb436da6a5292accde36e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:18:28.723901 9149 sys_catalog.cc:455] T 00000000000000000000000000000000 P eadb8a55f4cb436da6a5292accde36e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "eadb8a55f4cb436da6a5292accde36e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eadb8a55f4cb436da6a5292accde36e8" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 37613 } } }
I20260618 22:18:28.723901 9150 sys_catalog.cc:455] T 00000000000000000000000000000000 P eadb8a55f4cb436da6a5292accde36e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eadb8a55f4cb436da6a5292accde36e8. Latest consensus state: current_term: 2 leader_uuid: "eadb8a55f4cb436da6a5292accde36e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eadb8a55f4cb436da6a5292accde36e8" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 37613 } } }
I20260618 22:18:28.724036 9149 sys_catalog.cc:458] T 00000000000000000000000000000000 P eadb8a55f4cb436da6a5292accde36e8 [sys.catalog]: This master's current role is: LEADER
I20260618 22:18:28.724042 9150 sys_catalog.cc:458] T 00000000000000000000000000000000 P eadb8a55f4cb436da6a5292accde36e8 [sys.catalog]: This master's current role is: LEADER
I20260618 22:18:28.724366 9157 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:18:28.727396 9157 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:18:28.728178 9157 catalog_manager.cc:1259] Loaded cluster ID: b76aac40ea9340a795f096cf3aab0f7b
I20260618 22:18:28.728248 9157 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:18:28.730530 9157 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:18:28.731149 9157 catalog_manager.cc:5879] T 00000000000000000000000000000000 P eadb8a55f4cb436da6a5292accde36e8: Loaded TSK: 0
I20260618 22:18:28.732313 9157 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260618 22:18:28.807005 9146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:18:28.807273 9146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:18:28.807343 9146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:18:28.811283 9146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:18:28.811367 9146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:18:28.811475 9146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260618 22:18:28.816461 9146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:36987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/krb5kdc/oryx_127.0.98.65.keytab
--principal=oryx/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:46105
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=44569
--tserver_master_addrs=127.0.98.126:37613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:18:28.817387 9146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:18:28.818380 9146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:18:28.828816 9170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:28.828974 9146 server_base.cc:1034] running on GCE node
W20260618 22:18:28.828833 9171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:28.829418 9173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:28.829810 9146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:18:28.830457 9146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:18:28.831664 9146 hybrid_clock.cc:648] HybridClock initialized: now 1781821108831638 us; error 43 us; skew 500 ppm
Jun 18 22:18:28 dist-test-slave-g6r4 krb5kdc[8563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821108, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:18:28.834617 9146 init.cc:377] Logged in from keytab as oryx/127.0.98.65@KRBTEST.COM (short username oryx)
I20260618 22:18:28.835490 9146 webserver.cc:460] Webserver started at http://127.0.98.65:44569/ using document root <none> and password file <none>
I20260618 22:18:28.836066 9146 fs_manager.cc:359] Metadata directory not provided
I20260618 22:18:28.836171 9146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:18:28.840144 9146 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:18:28.842206 9180 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:28.843501 9146 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:18:28.843636 9146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-0/wal
uuid: "4c4e50c0891340a4a8b0f7bd425ea8f2"
format_stamp: "Formatted at 2026-06-18 22:18:27 on dist-test-slave-g6r4"
server_key: "d4b143f4663b9ab3ca774b3d8630d88f"
server_key_iv: "e6c54af835833f4e1075f2753995c537"
server_key_version: "encryptionkey@0"
I20260618 22:18:28.844244 9146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:18:28.855217 9146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:18:28.858474 9146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:18:28.858659 9146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:18:28.859196 9146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:18:28.860020 9146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:18:28.860090 9146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:28.860155 9146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:18:28.860203 9146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:28.869575 9146 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:46105
I20260618 22:18:28.869638 9293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:46105 every 8 connection(s)
I20260618 22:18:28.870599 9146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
I20260618 22:18:28.880270 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 9146
I20260618 22:18:28.880767 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:34785
--local_ip_for_outbound_sockets=127.0.98.66
--tserver_master_addrs=127.0.98.126:37613
--webserver_port=46707
--webserver_interface=127.0.98.66
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:36987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.66
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/krb5kdc/oryx_127.0.98.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 18 22:18:28 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:18:28 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:18:28.882975 9297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:28.872614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37613 (local address 127.0.98.65:38113)
0618 22:18:28.873059 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:28.873109 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:28.874463 (+ 1354us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:28.875571 (+ 1108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:28.875578 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:28.875905 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:28.876328 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:28.876340 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:28.877603 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:28.877606 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:28.878526 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:28.878536 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:28.879231 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:28.879234 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:28.879370 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:28.879916 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:28.879931 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:28.882297 (+ 2366us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:37613: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":139,"threads_started":1}
W20260618 22:18:28.883420 9296 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:18:28.872589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:38113 (local address 127.0.98.126:37613)
0618 22:18:28.873019 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:28.873046 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:28.873272 (+ 226us) server_negotiation.cc:407] Connection header received
0618 22:18:28.874879 (+ 1607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:28.874891 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:28.875204 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:28.875495 (+ 291us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:28.876527 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:28.877481 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:28.878641 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:28.879101 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:28.879230 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:28.882567 (+ 3337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.65:38113: BlockingRecv error: recv got EOF from 127.0.98.65:38113 (error 108)
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":128,"threads_started":1}
W20260618 22:18:28.883510 9294 heartbeater.cc:644] Failed to heartbeat to 127.0.98.126:37613 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.126:37613: Client connection negotiation failed: client connection to 127.0.98.126:37613: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Jun 18 22:18:28 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:18:28 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:18:28.889423 9297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:28.883980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37613 (local address 127.0.98.65:38465)
0618 22:18:28.884156 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:28.884169 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:28.884238 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:28.884460 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:28.884462 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:28.884579 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:28.884656 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:28.884660 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:28.885718 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:28.885720 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:28.886296 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:28.886305 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:28.886935 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:28.886937 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:28.887001 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:28.887506 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:28.887517 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:28.889328 (+ 1811us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:37613: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111}
W20260618 22:18:28.889649 9296 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:18:28.884085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:38465 (local address 127.0.98.126:37613)
0618 22:18:28.884231 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:28.884237 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:28.884254 (+ 17us) server_negotiation.cc:407] Connection header received
0618 22:18:28.884296 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:28.884302 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:28.884354 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:28.884443 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:28.884761 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:28.885616 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:28.886416 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:28.886824 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:28.886953 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:28.889508 (+ 2555us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.65:38465: BlockingRecv error: recv got EOF from 127.0.98.65:38465 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 18 22:18:28 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:18:28 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:18:28.895244 9297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:28.890040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37613 (local address 127.0.98.65:37287)
0618 22:18:28.890160 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:28.890184 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:28.890265 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:28.890502 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:28.890508 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:28.890684 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:28.890792 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:28.890802 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:28.891471 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:28.891475 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:28.892438 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:28.892451 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:28.892848 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:28.892852 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:28.892934 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:28.893338 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:28.893352 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:28.895123 (+ 1771us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:37613: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":31}
W20260618 22:18:28.895375 9296 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:18:28.890099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:37287 (local address 127.0.98.126:37613)
0618 22:18:28.890205 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:28.890208 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:28.890223 (+ 15us) server_negotiation.cc:407] Connection header received
0618 22:18:28.890359 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:28.890361 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:28.890393 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:28.890456 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:28.890907 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:28.891362 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:28.892553 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:28.892745 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:28.892815 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:28.895231 (+ 2416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.65:37287: BlockingRecv error: recv got EOF from 127.0.98.65:37287 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260618 22:18:28.895666 9294 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260618 22:18:28.996335 9298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:18:28.996567 9298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:18:28.996644 9298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:18:29.000720 9298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:18:29.000770 9298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:18:29.000876 9298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260618 22:18:29.005884 9298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:36987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/krb5kdc/oryx_127.0.98.66.keytab
--principal=oryx/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:34785
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=46707
--tserver_master_addrs=127.0.98.126:37613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:18:29.006840 9298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:18:29.007833 9298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:18:29.016992 9303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:29.017001 9306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:29.017019 9304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:29.017072 9298 server_base.cc:1034] running on GCE node
I20260618 22:18:29.017596 9298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:18:29.018252 9298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:18:29.019440 9298 hybrid_clock.cc:648] HybridClock initialized: now 1781821109019426 us; error 30 us; skew 500 ppm
Jun 18 22:18:29 dist-test-slave-g6r4 krb5kdc[8563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821109, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:18:29.022217 9298 init.cc:377] Logged in from keytab as oryx/127.0.98.66@KRBTEST.COM (short username oryx)
I20260618 22:18:29.023111 9298 webserver.cc:460] Webserver started at http://127.0.98.66:46707/ using document root <none> and password file <none>
I20260618 22:18:29.023664 9298 fs_manager.cc:359] Metadata directory not provided
I20260618 22:18:29.023815 9298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:18:29.027549 9298 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:18:29.029536 9313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:29.030860 9298 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:18:29.031000 9298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-1/wal
uuid: "fbc3e88ddcc1435ab0ebf3cf49928862"
format_stamp: "Formatted at 2026-06-18 22:18:28 on dist-test-slave-g6r4"
server_key: "0a64f9b738bf30a6fae05951cb09bf6c"
server_key_iv: "6038ef8fe169cb4e0381b340424123fd"
server_key_version: "encryptionkey@0"
I20260618 22:18:29.031514 9298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:18:29.048545 9298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:18:29.051349 9298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:18:29.051542 9298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:18:29.052093 9298 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:18:29.052893 9298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:18:29.052938 9298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:29.053004 9298 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:18:29.053045 9298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:29.062117 9298 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:34785
I20260618 22:18:29.062140 9426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:34785 every 8 connection(s)
I20260618 22:18:29.063083 9298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
I20260618 22:18:29.068107 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 9298
I20260618 22:18:29.068620 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:41329
--local_ip_for_outbound_sockets=127.0.98.67
--tserver_master_addrs=127.0.98.126:37613
--webserver_port=43495
--webserver_interface=127.0.98.67
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:36987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.67
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/krb5kdc/oryx_127.0.98.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 18 22:18:29 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:18:29 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
W20260618 22:18:29.074561 9296 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:18:29.064930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:57161 (local address 127.0.98.126:37613)
0618 22:18:29.065057 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:29.065060 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:29.065890 (+ 830us) server_negotiation.cc:407] Connection header received
0618 22:18:29.067199 (+ 1309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:29.067202 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:29.067249 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:29.067326 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:29.068465 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:29.069289 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:29.070555 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:29.070938 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:29.071054 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:29.074389 (+ 3335us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.66:57161: BlockingRecv error: recv got EOF from 127.0.98.66:57161 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
I20260618 22:18:29.074980 9429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:29.065262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37613 (local address 127.0.98.66:57161)
0618 22:18:29.065694 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:29.065735 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:29.067004 (+ 1269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:29.067450 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:29.067457 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:29.067794 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:29.068317 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:29.068333 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:29.069443 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:29.069446 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:29.070424 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:29.070434 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:29.071092 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:29.071095 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:29.071234 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:29.071838 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:29.071853 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:29.074191 (+ 2338us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:37613: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":115,"threads_started":1}
W20260618 22:18:29.075575 9427 heartbeater.cc:644] Failed to heartbeat to 127.0.98.126:37613 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.126:37613: Client connection negotiation failed: client connection to 127.0.98.126:37613: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Jun 18 22:18:29 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:18:29 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:18:29.081725 9429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:29.076035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37613 (local address 127.0.98.66:43669)
0618 22:18:29.076144 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:29.076158 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:29.076251 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:29.076510 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:29.076512 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:29.076639 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:29.076761 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:29.076766 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:29.077964 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:29.077967 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:29.078501 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:29.078509 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:29.079213 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:29.079217 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:29.079291 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:29.079878 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:29.079889 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:29.081628 (+ 1739us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:37613: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":46}
W20260618 22:18:29.081883 9296 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:18:29.076126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:43669 (local address 127.0.98.126:37613)
0618 22:18:29.076265 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:29.076272 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:29.076288 (+ 16us) server_negotiation.cc:407] Connection header received
0618 22:18:29.076330 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:29.076336 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:29.076403 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:29.076500 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:29.076872 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:29.077773 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:29.078616 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:29.079058 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:29.079247 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:29.081773 (+ 2526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.66:43669: BlockingRecv error: recv got EOF from 127.0.98.66:43669 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 18 22:18:29 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:18:29 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:18:29.087440 9429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:29.082442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37613 (local address 127.0.98.66:41029)
0618 22:18:29.082527 (+ 85us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:29.082543 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:29.082620 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:29.082859 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:29.082861 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:29.082974 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:29.083035 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:29.083039 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:29.084048 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:29.084051 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:29.084568 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:29.084576 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:29.085138 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:29.085141 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:29.085208 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:29.085700 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:29.085742 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:29.087353 (+ 1611us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:37613: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":24}
W20260618 22:18:29.087618 9296 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:18:29.082509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:41029 (local address 127.0.98.126:37613)
0618 22:18:29.082603 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:29.082607 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:29.082621 (+ 14us) server_negotiation.cc:407] Connection header received
0618 22:18:29.082717 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:29.082720 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:29.082761 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:29.082841 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:29.083133 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:29.083897 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:29.084675 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:29.085017 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:29.085114 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:29.087486 (+ 2372us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.66:41029: BlockingRecv error: recv got EOF from 127.0.98.66:41029 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260618 22:18:29.087812 9427 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260618 22:18:29.179930 9430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:18:29.180153 9430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:18:29.180193 9430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:18:29.184158 9430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:18:29.184216 9430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:18:29.184271 9430 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260618 22:18:29.189333 9430 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:36987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/krb5kdc/oryx_127.0.98.67.keytab
--principal=oryx/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:41329
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=43495
--tserver_master_addrs=127.0.98.126:37613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9430
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:18:29.190354 9430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:18:29.191352 9430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:18:29.200567 9435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:29.200578 9438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:29.200779 9436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:29.200861 9430 server_base.cc:1034] running on GCE node
I20260618 22:18:29.201318 9430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:18:29.201982 9430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:18:29.203161 9430 hybrid_clock.cc:648] HybridClock initialized: now 1781821109203143 us; error 29 us; skew 500 ppm
Jun 18 22:18:29 dist-test-slave-g6r4 krb5kdc[8563](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821109, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:18:29.206146 9430 init.cc:377] Logged in from keytab as oryx/127.0.98.67@KRBTEST.COM (short username oryx)
I20260618 22:18:29.206991 9430 webserver.cc:460] Webserver started at http://127.0.98.67:43495/ using document root <none> and password file <none>
I20260618 22:18:29.207484 9430 fs_manager.cc:359] Metadata directory not provided
I20260618 22:18:29.207556 9430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:18:29.211431 9430 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:18:29.213483 9445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:29.214792 9430 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:18:29.214927 9430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-2/wal
uuid: "81e11ef0cf8d49e4bcd65052106f9cdd"
format_stamp: "Formatted at 2026-06-18 22:18:28 on dist-test-slave-g6r4"
server_key: "197485229128572163be4942ba6db711"
server_key_iv: "f99e9705b28ba8cd7e7c8f200f54796a"
server_key_version: "encryptionkey@0"
I20260618 22:18:29.215462 9430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:18:29.236606 9430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:18:29.239843 9430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:18:29.240061 9430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:18:29.240697 9430 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:18:29.241509 9430 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:18:29.241564 9430 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:29.241628 9430 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:18:29.241660 9430 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:29.250622 9430 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:41329
I20260618 22:18:29.250694 9558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:41329 every 8 connection(s)
I20260618 22:18:29.251612 9430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
I20260618 22:18:29.255321 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 9430
Jun 18 22:18:29 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:18:29 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
W20260618 22:18:29.263566 9296 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:18:29.253469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:41661 (local address 127.0.98.126:37613)
0618 22:18:29.253596 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:29.253600 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:29.254586 (+ 986us) server_negotiation.cc:407] Connection header received
0618 22:18:29.256015 (+ 1429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:29.256019 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:29.256080 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:29.256190 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:29.257447 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:29.258093 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:29.259188 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:29.259393 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:29.259495 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:29.263405 (+ 3910us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.67:41661: BlockingRecv error: recv got EOF from 127.0.98.67:41661 (error 108)
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":49}
I20260618 22:18:29.264065 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:29.253945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37613 (local address 127.0.98.67:41661)
0618 22:18:29.254436 (+ 491us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:29.254478 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:29.255766 (+ 1288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:29.256503 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:29.256510 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:29.256857 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:29.257312 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:29.257324 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:29.258210 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:29.258213 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:29.259050 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:29.259059 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:29.259527 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:29.259531 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:29.259695 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:29.260203 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:29.260220 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:29.263219 (+ 2999us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:37613: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":142,"threads_started":1}
W20260618 22:18:29.264536 9559 heartbeater.cc:644] Failed to heartbeat to 127.0.98.126:37613 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.126:37613: Client connection negotiation failed: client connection to 127.0.98.126:37613: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
I20260618 22:18:29.266007 9563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:29.256306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33654 (local address 127.0.98.126:37613)
0618 22:18:29.256539 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:29.256542 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:29.256551 (+ 9us) server_negotiation.cc:407] Connection header received
0618 22:18:29.256581 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:29.256583 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:29.256615 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:29.256715 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:29.257414 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:29.258127 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:29.258985 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:29.259209 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:29.259318 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:29.260817 (+ 1499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:29.260824 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:29.260841 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:29.260879 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:29.263250 (+ 2371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:29.263883 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:29.263890 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:29.263894 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:29.264016 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:29.264356 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:29.264362 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:29.264366 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:29.264844 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:29.265068 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:29.265533 (+ 465us) server_negotiation.cc:299] Negotiation successful
0618 22:18:29.265757 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":75,"server-negotiator.queue_time_us":164,"thread_start_us":69,"threads_started":1}
Jun 18 22:18:29 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:18:29 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:18:29.270634 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:29.265057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37613 (local address 127.0.98.67:38953)
0618 22:18:29.265164 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:29.265180 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:29.265252 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:29.265486 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:29.265489 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:29.265643 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:29.265749 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:29.265756 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:29.266836 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:29.266838 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:29.267369 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:29.267376 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:29.268028 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:29.268032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:29.268132 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:29.268604 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:29.268618 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:29.270548 (+ 1930us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:37613: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":44}
W20260618 22:18:29.270814 9296 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:18:29.265105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:38953 (local address 127.0.98.126:37613)
0618 22:18:29.265237 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:29.265242 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:29.265256 (+ 14us) server_negotiation.cc:407] Connection header received
0618 22:18:29.265318 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:29.265324 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:29.265379 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:29.265458 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:29.265850 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:29.266680 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:29.267508 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:29.267787 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:29.267903 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:29.270692 (+ 2789us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.67:38953: BlockingRecv error: recv got EOF from 127.0.98.67:38953 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 18 22:18:29 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:18:29 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:18:29.275497 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:29.271246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37613 (local address 127.0.98.67:32871)
0618 22:18:29.271332 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:29.271347 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:29.271415 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:29.271581 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:29.271583 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:29.271696 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:29.271759 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:29.271762 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:29.272385 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:29.272386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:29.272882 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:29.272889 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:29.273289 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:29.273291 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:29.273357 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:29.273834 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:29.273858 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:29.275416 (+ 1558us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:37613: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":25}
W20260618 22:18:29.275669 9296 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:18:29.271300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:32871 (local address 127.0.98.126:37613)
0618 22:18:29.271421 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:29.271424 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:29.271435 (+ 11us) server_negotiation.cc:407] Connection header received
0618 22:18:29.271463 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:29.271465 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:29.271495 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:29.271556 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:29.271848 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:29.272302 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:29.272980 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:29.273181 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:29.273266 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:29.275523 (+ 2257us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.67:32871: BlockingRecv error: recv got EOF from 127.0.98.67:32871 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260618 22:18:29.275755 9559 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 18 22:18:29 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:18:29 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:18:29.902124 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:29.896345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37613 (local address 127.0.98.65:42909)
0618 22:18:29.896602 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:29.896616 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:29.896763 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:29.897095 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:29.897099 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:29.897330 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:29.897528 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:29.897537 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:29.898471 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:29.898474 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:29.899017 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:29.899024 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:29.899482 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:29.899484 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:29.899552 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:29.900090 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:29.900111 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:29.902009 (+ 1898us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:37613: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":89,"threads_started":1}
W20260618 22:18:29.902290 9565 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:18:29.896402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:42909 (local address 127.0.98.126:37613)
0618 22:18:29.896695 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:29.896699 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:29.896713 (+ 14us) server_negotiation.cc:407] Connection header received
0618 22:18:29.896867 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:29.896870 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:29.896938 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:29.897047 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:29.897714 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:29.898371 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:29.899142 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:29.899385 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:29.899470 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:29.902133 (+ 2663us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.65:42909: BlockingRecv error: recv got EOF from 127.0.98.65:42909 (error 108)
Metrics: {"server-negotiator.queue_time_us":220,"thread_start_us":94,"threads_started":1}
Jun 18 22:18:30 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:18:30 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:18:30.093803 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:30.088449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37613 (local address 127.0.98.66:42405)
0618 22:18:30.088728 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:30.088745 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:30.088887 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:30.089147 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:30.089149 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:30.089356 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:30.089460 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:30.089466 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:30.090289 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:30.090296 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:30.090849 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:30.090857 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:30.091302 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:30.091306 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:30.091399 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:30.091874 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:30.091885 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:30.093631 (+ 1746us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:37613: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":82,"threads_started":1}
W20260618 22:18:30.093922 9565 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:18:30.088503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:42405 (local address 127.0.98.126:37613)
0618 22:18:30.088699 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:30.088704 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:30.088833 (+ 129us) server_negotiation.cc:407] Connection header received
0618 22:18:30.088985 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:30.088988 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:30.089040 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:30.089126 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:30.089578 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:30.090180 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:30.090967 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:30.091192 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:30.091302 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:30.093789 (+ 2487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.66:42405: BlockingRecv error: recv got EOF from 127.0.98.66:42405 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Jun 18 22:18:30 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:18:30 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:18:30.282771 9567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:30.276495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37613 (local address 127.0.98.67:60871)
0618 22:18:30.276755 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:30.276769 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:30.276950 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:30.277220 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:30.277223 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:30.277434 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:30.277521 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:30.277527 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:30.278714 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:30.278721 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:30.279623 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:30.279631 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:30.280086 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:30.280090 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:30.280174 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:30.280727 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:30.280741 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:30.282596 (+ 1855us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:37613: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":65,"threads_started":1}
W20260618 22:18:30.282810 9565 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:18:30.276549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:60871 (local address 127.0.98.126:37613)
0618 22:18:30.276693 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:30.276697 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:30.276851 (+ 154us) server_negotiation.cc:407] Connection header received
0618 22:18:30.277051 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:30.277055 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:30.277108 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:30.277197 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:30.277668 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:30.278578 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:30.279736 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:30.279978 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:30.280087 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:30.282708 (+ 2621us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.67:60871: BlockingRecv error: recv got EOF from 127.0.98.67:60871 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 18 22:18:30 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:18:30 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:18:30.908828 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:30.902922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37613 (local address 127.0.98.65:56231)
0618 22:18:30.903397 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:30.903414 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:30.903530 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:30.903773 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:30.903777 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:30.903996 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:30.904138 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:30.904147 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:30.905235 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:30.905239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:30.905857 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:30.905866 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:30.906340 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:30.906342 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:30.906407 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:30.906861 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:30.906872 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:30.908644 (+ 1772us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:37613: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":108,"threads_started":1}
W20260618 22:18:30.908876 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:18:30.903047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:56231 (local address 127.0.98.126:37613)
0618 22:18:30.903320 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:30.903325 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:30.903499 (+ 174us) server_negotiation.cc:407] Connection header received
0618 22:18:30.903609 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:30.903611 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:30.903652 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:30.903753 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:30.904295 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:30.905124 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:30.906010 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:30.906240 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:30.906323 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:30.908763 (+ 2440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.65:56231: BlockingRecv error: recv got EOF from 127.0.98.65:56231 (error 108)
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":94,"threads_started":1}
Jun 18 22:18:31 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:18:31 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:18:31.099649 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:31.094606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37613 (local address 127.0.98.66:33861)
0618 22:18:31.094882 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:31.094899 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:31.094991 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:31.095199 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:31.095201 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:31.095384 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:31.095484 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:31.095489 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:31.096173 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:31.096175 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:31.096740 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:31.096749 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:31.097182 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:31.097185 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:31.097271 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:31.097749 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:31.097763 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:31.099516 (+ 1753us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:37613: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":107,"threads_started":1}
W20260618 22:18:31.099725 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:18:31.094635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:33861 (local address 127.0.98.126:37613)
0618 22:18:31.094800 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:31.094809 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:31.094967 (+ 158us) server_negotiation.cc:407] Connection header received
0618 22:18:31.095055 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:31.095056 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:31.095097 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:31.095164 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:31.095578 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:31.096082 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:31.096838 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:31.097059 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:31.097134 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:31.099636 (+ 2502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.66:33861: BlockingRecv error: recv got EOF from 127.0.98.66:33861 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Jun 18 22:18:31 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:18:31 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:18:31.289624 9571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:31.283667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37613 (local address 127.0.98.67:46173)
0618 22:18:31.283948 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:31.283962 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:31.284051 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:31.284279 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:31.284281 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:31.284453 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:31.284547 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:31.284552 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:31.285354 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:31.285356 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:31.285927 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:31.285935 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:31.286435 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:31.286437 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:31.286499 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:31.287057 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:31.287069 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:31.289413 (+ 2344us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:37613: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":100,"threads_started":1}
W20260618 22:18:31.289736 9569 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:18:31.283886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:46173 (local address 127.0.98.126:37613)
0618 22:18:31.284040 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:31.284044 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:31.284058 (+ 14us) server_negotiation.cc:407] Connection header received
0618 22:18:31.284124 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:31.284127 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:31.284170 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:31.284259 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:31.284656 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:31.285255 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:31.286050 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:31.286333 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:31.286422 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:31.289576 (+ 3154us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.67:46173: BlockingRecv error: recv got EOF from 127.0.98.67:46173 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 18 22:18:31 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:18:31 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:18:31.915122 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:31.909603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37613 (local address 127.0.98.65:32957)
0618 22:18:31.909971 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:31.909987 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:31.910104 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:31.910415 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:31.910418 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:31.910656 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:31.910777 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:31.910784 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:31.911540 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:31.911542 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:31.912092 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:31.912099 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:31.912510 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:31.912512 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:31.912570 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:31.913118 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:31.913138 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:31.914996 (+ 1858us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:37613: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":77,"threads_started":1}
W20260618 22:18:31.915274 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:18:31.909665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:32957 (local address 127.0.98.126:37613)
0618 22:18:31.909973 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:31.909978 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:31.910056 (+ 78us) server_negotiation.cc:407] Connection header received
0618 22:18:31.910200 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:31.910220 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:31.910277 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:31.910369 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:31.910906 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:31.911445 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:31.912194 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:31.912414 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:31.912493 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:31.915123 (+ 2630us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.65:32957: BlockingRecv error: recv got EOF from 127.0.98.65:32957 (error 108)
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":62,"threads_started":1}
Jun 18 22:18:32 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:18:32 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:18:32.106084 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:32.100431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37613 (local address 127.0.98.66:46561)
0618 22:18:32.100759 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:32.100780 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:32.100886 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:32.101124 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:32.101126 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:32.101299 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:32.101398 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:32.101402 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:32.102206 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:32.102209 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:32.102747 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:32.102755 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:32.103171 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:32.103173 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:32.103286 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:32.103852 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:32.103890 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:32.105878 (+ 1988us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:37613: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":106,"threads_started":1}
W20260618 22:18:32.106114 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:18:32.100550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:46561 (local address 127.0.98.126:37613)
0618 22:18:32.100710 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:32.100714 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:32.100852 (+ 138us) server_negotiation.cc:407] Connection header received
0618 22:18:32.100972 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:32.100974 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:32.101015 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:32.101087 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:32.101508 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:32.102097 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:32.102859 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:32.103074 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:32.103165 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:32.106027 (+ 2862us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.66:46561: BlockingRecv error: recv got EOF from 127.0.98.66:46561 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Jun 18 22:18:32 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:18:32 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:18:32.296746 9575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:32.290408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37613 (local address 127.0.98.67:50087)
0618 22:18:32.290817 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:32.290840 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:32.290981 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:32.291296 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:32.291299 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:32.291517 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:32.291654 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:32.291661 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:32.292504 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:32.292510 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:32.293075 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:32.293082 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:32.293540 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:32.293543 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:32.293628 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:32.294132 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:32.294144 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:32.296599 (+ 2455us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:37613: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":87,"threads_started":1}
W20260618 22:18:32.296838 9573 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:18:32.290555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:50087 (local address 127.0.98.126:37613)
0618 22:18:32.290734 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:32.290740 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:32.290924 (+ 184us) server_negotiation.cc:407] Connection header received
0618 22:18:32.291109 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:32.291129 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:32.291177 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:32.291273 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:32.291844 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:32.292390 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:32.293197 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:32.293427 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:32.293536 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:32.296723 (+ 3187us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.67:50087: BlockingRecv error: recv got EOF from 127.0.98.67:50087 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Jun 18 22:18:32 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:18:32 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:18:32.926806 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:32.915861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37613 (local address 127.0.98.65:50221)
0618 22:18:32.916118 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:32.916132 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:32.916231 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:32.916467 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:32.916470 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:32.916635 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:32.916729 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:32.916734 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:32.917455 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:32.917458 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:32.918195 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:32.918206 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:32.918845 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:32.918849 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:32.918938 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:32.924847 (+ 5909us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:32.924859 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:32.926686 (+ 1827us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:37613: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":85,"threads_started":1}
W20260618 22:18:32.927031 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:18:32.915929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:50221 (local address 127.0.98.126:37613)
0618 22:18:32.916185 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:32.916189 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:32.916202 (+ 13us) server_negotiation.cc:407] Connection header received
0618 22:18:32.916316 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:32.916318 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:32.916361 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:32.916439 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:32.916819 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:32.917339 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:32.918333 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:32.918714 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:32.918828 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:32.926869 (+ 8041us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.65:50221: BlockingRecv error: recv got EOF from 127.0.98.65:50221 (error 108)
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":90,"threads_started":1}
Jun 18 22:18:33 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:18:33 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:18:33.112399 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:33.106850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37613 (local address 127.0.98.66:57023)
0618 22:18:33.107206 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:33.107222 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:33.107364 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:33.107625 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:33.107628 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:33.107826 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:33.107913 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:33.107918 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:33.108740 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:33.108742 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:33.109298 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:33.109305 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:33.109752 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:33.109756 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:33.109842 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:33.110466 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:33.110486 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:33.112275 (+ 1789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:37613: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":102,"threads_started":1}
W20260618 22:18:33.112627 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:18:33.106994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:57023 (local address 127.0.98.126:37613)
0618 22:18:33.107195 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:33.107201 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:33.107302 (+ 101us) server_negotiation.cc:407] Connection header received
0618 22:18:33.107461 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:33.107465 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:33.107517 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:33.107605 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:33.108024 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:33.108576 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:33.109411 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:33.109638 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:33.109748 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:33.112409 (+ 2661us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.66:57023: BlockingRecv error: recv got EOF from 127.0.98.66:57023 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Jun 18 22:18:33 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:18:33 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:18:33.303216 9579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:33.297496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37613 (local address 127.0.98.67:36477)
0618 22:18:33.297774 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:33.297786 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:33.297946 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:33.298199 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:33.298202 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:33.298406 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:33.298493 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:33.298499 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:33.299249 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:33.299254 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:33.299809 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:33.299816 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:33.300276 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:33.300279 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:33.300363 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:33.300908 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:33.300920 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:33.303086 (+ 2166us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:37613: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":85,"threads_started":1}
W20260618 22:18:33.303375 9577 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:18:33.297561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:36477 (local address 127.0.98.126:37613)
0618 22:18:33.297717 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:33.297722 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:33.297859 (+ 137us) server_negotiation.cc:407] Connection header received
0618 22:18:33.298044 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:33.298047 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:33.298095 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:33.298180 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:33.298613 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:33.299139 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:33.299932 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:33.300168 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:33.300274 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:33.303213 (+ 2939us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.67:36477: BlockingRecv error: recv got EOF from 127.0.98.67:36477 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Jun 18 22:18:33 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:18:33 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:18:33.933056 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:33.927635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37613 (local address 127.0.98.65:50437)
0618 22:18:33.927921 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:33.927938 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:33.928056 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:33.928346 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:33.928349 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:33.928554 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:33.928641 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:33.928648 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:33.929387 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:33.929390 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:33.930021 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:33.930030 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:33.930487 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:33.930491 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:33.930575 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:33.931106 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:33.931127 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:33.932880 (+ 1753us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:37613: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":82,"threads_started":1}
W20260618 22:18:33.933118 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:18:33.927697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:50437 (local address 127.0.98.126:37613)
0618 22:18:33.927922 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:33.927927 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:33.928021 (+ 94us) server_negotiation.cc:407] Connection header received
0618 22:18:33.928165 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:33.928168 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:33.928227 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:33.928321 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:33.928755 (+ 434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:33.929282 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:33.930141 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:33.930377 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:33.930485 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:33.932995 (+ 2510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.65:50437: BlockingRecv error: recv got EOF from 127.0.98.65:50437 (error 108)
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":65,"threads_started":1}
Jun 18 22:18:34 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:18:34 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:18:34.118911 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:34.113286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37613 (local address 127.0.98.66:46949)
0618 22:18:34.113646 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:34.113663 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:34.113872 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:34.114160 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:34.114162 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:34.114384 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:34.114486 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:34.114491 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:34.115235 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:34.115238 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:34.115809 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:34.115816 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:34.116262 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:34.116266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:34.116350 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:34.116889 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:34.116901 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:34.118781 (+ 1880us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:37613: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":130,"threads_started":1}
W20260618 22:18:34.119093 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:18:34.113472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:46949 (local address 127.0.98.126:37613)
0618 22:18:34.113650 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:34.113658 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:34.113802 (+ 144us) server_negotiation.cc:407] Connection header received
0618 22:18:34.113996 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:34.113999 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:34.114050 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:34.114135 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:34.114597 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:34.115122 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:34.115922 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:34.116149 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:34.116258 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:34.118926 (+ 2668us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.66:46949: BlockingRecv error: recv got EOF from 127.0.98.66:46949 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
Jun 18 22:18:34 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:18:34 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:18:34.309719 9583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:34.304094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37613 (local address 127.0.98.67:49793)
0618 22:18:34.304367 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:34.304381 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:34.304474 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:34.304747 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:34.304749 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:34.304927 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:34.305017 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:34.305022 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:34.305712 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:34.305714 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:34.306312 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:34.306321 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:34.306880 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:34.306885 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:34.306975 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:34.307620 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:34.307641 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:34.309549 (+ 1908us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:37613: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":96,"threads_started":1}
W20260618 22:18:34.309787 9581 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:18:34.304233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:49793 (local address 127.0.98.126:37613)
0618 22:18:34.304385 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:34.304389 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:34.304453 (+ 64us) server_negotiation.cc:407] Connection header received
0618 22:18:34.304555 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:34.304557 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:34.304600 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:34.304672 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:34.305116 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:34.305618 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:34.306423 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:34.306760 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:34.306885 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:34.309682 (+ 2797us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.67:49793: BlockingRecv error: recv got EOF from 127.0.98.67:49793 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Jun 18 22:18:34 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:18:34 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:18:34.939896 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:34.933902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37613 (local address 127.0.98.65:49819)
0618 22:18:34.934223 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:34.934237 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:34.934330 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:34.934596 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:34.934598 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:34.934771 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:34.934862 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:34.934904 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:34.935786 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:34.935789 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:34.936406 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:34.936419 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:34.937031 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:34.937037 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:34.937129 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:34.937757 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:34.937768 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:34.939694 (+ 1926us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:37613: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":105,"threads_started":1}
W20260618 22:18:34.939949 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:18:34.933968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:49819 (local address 127.0.98.126:37613)
0618 22:18:34.934199 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:34.934203 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:34.934302 (+ 99us) server_negotiation.cc:407] Connection header received
0618 22:18:34.934407 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:34.934409 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:34.934452 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:34.934564 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:34.934996 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:34.935669 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:34.936531 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:34.936912 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:34.936999 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:34.939819 (+ 2820us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.65:49819: BlockingRecv error: recv got EOF from 127.0.98.65:49819 (error 108)
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":73,"threads_started":1}
Jun 18 22:18:35 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:18:35 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:18:35.125375 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:35.119605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37613 (local address 127.0.98.66:37613)
0618 22:18:35.119856 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:35.119882 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:35.119984 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:35.120300 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:35.120303 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:35.120556 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:35.120660 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:35.120666 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:35.121534 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:35.121537 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:35.122116 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:35.122125 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:35.122591 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:35.122594 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:35.122656 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:35.123195 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:35.123215 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:35.125207 (+ 1992us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:37613: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":89,"threads_started":1}
W20260618 22:18:35.125437 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:18:35.119753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:37613 (local address 127.0.98.126:37613)
0618 22:18:35.119926 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:35.119930 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:35.119948 (+ 18us) server_negotiation.cc:407] Connection header received
0618 22:18:35.120096 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:35.120107 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:35.120165 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:35.120245 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:35.120764 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:35.121437 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:35.122251 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:35.122489 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:35.122569 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:35.125331 (+ 2762us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.66:37613: BlockingRecv error: recv got EOF from 127.0.98.66:37613 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 18 22:18:35 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:18:35 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:18:35.316291 9587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:35.310552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37613 (local address 127.0.98.67:46949)
0618 22:18:35.310897 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:35.310910 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:35.311049 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:35.311320 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:35.311322 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:35.311532 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:35.311620 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:35.311626 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:35.312375 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:35.312381 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:35.312949 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:35.312957 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:35.313444 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:35.313448 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:35.313529 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:35.314128 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:35.314139 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:35.316120 (+ 1981us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:37613: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":150,"threads_started":1}
W20260618 22:18:35.316344 9584 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:18:35.310625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:46949 (local address 127.0.98.126:37613)
0618 22:18:35.310796 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:35.310800 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:35.310989 (+ 189us) server_negotiation.cc:407] Connection header received
0618 22:18:35.311147 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:35.311151 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:35.311205 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:35.311295 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:35.311739 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:35.312264 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:35.313070 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:35.313292 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:35.313436 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:35.316233 (+ 2797us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.67:46949: BlockingRecv error: recv got EOF from 127.0.98.67:46949 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 18 22:18:35 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:18:35 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:18:35.946635 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:35.940773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37613 (local address 127.0.98.65:48999)
0618 22:18:35.941115 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:35.941135 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:35.941258 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:35.941521 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:35.941524 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:35.941754 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:35.941861 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:35.941867 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:35.942757 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:35.942760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:35.943389 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:35.943399 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:35.943950 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:35.943954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:35.944025 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:35.944605 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:35.944616 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:35.946523 (+ 1907us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:37613: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":70,"threads_started":1}
W20260618 22:18:35.946852 9589 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:18:35.940848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:48999 (local address 127.0.98.126:37613)
0618 22:18:35.941127 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:35.941131 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:35.941199 (+ 68us) server_negotiation.cc:407] Connection header received
0618 22:18:35.941350 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:35.941352 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:35.941395 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:35.941470 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:35.942003 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:35.942565 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:35.943501 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:35.943814 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:35.943921 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:35.946649 (+ 2728us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.65:48999: BlockingRecv error: recv got EOF from 127.0.98.65:48999 (error 108)
Metrics: {"server-negotiator.queue_time_us":213,"thread_start_us":111,"threads_started":1}
Jun 18 22:18:36 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:18:36 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:18:36.131762 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:36.126431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37613 (local address 127.0.98.66:46615)
0618 22:18:36.126819 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:36.126834 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:36.126937 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:36.127179 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:36.127181 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:36.127374 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:36.127473 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:36.127479 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:36.128245 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:36.128248 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:36.128849 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:36.128858 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:36.129253 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:36.129256 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:36.129315 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:36.129789 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:36.129801 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:36.131637 (+ 1836us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:37613: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":85,"threads_started":1}
W20260618 22:18:36.131934 9589 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:18:36.126570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:46615 (local address 127.0.98.126:37613)
0618 22:18:36.126746 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:36.126750 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:36.126904 (+ 154us) server_negotiation.cc:407] Connection header received
0618 22:18:36.127016 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:36.127018 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:36.127058 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:36.127124 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:36.127576 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:36.128125 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:36.128955 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:36.129164 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:36.129239 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:36.131806 (+ 2567us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.66:46615: BlockingRecv error: recv got EOF from 127.0.98.66:46615 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Jun 18 22:18:36 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:18:36 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:18:36.322988 9591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:36.317185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37613 (local address 127.0.98.67:37185)
0618 22:18:36.317605 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:36.317619 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:36.317710 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:36.318021 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:36.318024 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:36.318249 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:36.318389 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:36.318401 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:36.319234 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:36.319236 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:36.319883 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:36.319893 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:36.320361 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:36.320365 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:36.320468 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:36.320942 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:36.320953 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:36.322881 (+ 1928us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:37613: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":83,"threads_started":1}
W20260618 22:18:36.323136 9589 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:18:36.317300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:37185 (local address 127.0.98.126:37613)
0618 22:18:36.317429 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:36.317434 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:36.317694 (+ 260us) server_negotiation.cc:407] Connection header received
0618 22:18:36.317782 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:36.317785 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:36.317847 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:36.317989 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:36.318519 (+ 530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:36.319135 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:36.320032 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:36.320250 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:36.320361 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:36.323018 (+ 2657us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.67:37185: BlockingRecv error: recv got EOF from 127.0.98.67:37185 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 18 22:18:36 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:18:36 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:18:36.953822 9592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:36.947407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37613 (local address 127.0.98.65:60509)
0618 22:18:36.947657 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:36.947672 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:36.947818 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:36.948153 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:36.948155 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:36.948351 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:36.948479 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:36.948486 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:36.949297 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:36.949300 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:36.949982 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:36.949992 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:36.950666 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:36.950669 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:36.950830 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:36.951496 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:36.951512 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:36.953709 (+ 2197us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:37613: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":87,"threads_started":1}
W20260618 22:18:36.954013 9593 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:18:36.947567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:60509 (local address 127.0.98.126:37613)
0618 22:18:36.947793 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:36.947796 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:36.947809 (+ 13us) server_negotiation.cc:407] Connection header received
0618 22:18:36.947910 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:36.947916 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:36.947968 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:36.948039 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:36.948588 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:36.949181 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:36.950178 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:36.950511 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:36.950627 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:36.953850 (+ 3223us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.65:60509: BlockingRecv error: recv got EOF from 127.0.98.65:60509 (error 108)
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":78,"threads_started":1}
Jun 18 22:18:37 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:18:37 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:18:37.138494 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:37.132627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37613 (local address 127.0.98.66:49003)
0618 22:18:37.132973 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:37.132984 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:37.133111 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:37.133388 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:37.133391 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:37.133600 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:37.133687 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:37.133692 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:37.134686 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:37.134692 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:37.135294 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:37.135301 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:37.135843 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:37.135847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:37.135934 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:37.136490 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:37.136506 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:37.138326 (+ 1820us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:37613: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":86,"threads_started":1}
W20260618 22:18:37.138531 9593 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:18:37.132694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:49003 (local address 127.0.98.126:37613)
0618 22:18:37.132850 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:37.132856 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:37.133061 (+ 205us) server_negotiation.cc:407] Connection header received
0618 22:18:37.133219 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:37.133223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:37.133273 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:37.133368 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:37.133853 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:37.134575 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:37.135409 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:37.135712 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:37.135840 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:37.138436 (+ 2596us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.66:49003: BlockingRecv error: recv got EOF from 127.0.98.66:49003 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 18 22:18:37 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:18:37 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:18:37.329058 9595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:37.323720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37613 (local address 127.0.98.67:37961)
0618 22:18:37.323992 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:37.324008 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:37.324172 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:37.324450 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:37.324453 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:37.324652 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:37.324739 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:37.324744 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:37.325509 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:37.325515 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:37.326090 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:37.326098 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:37.326547 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:37.326551 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:37.326634 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:37.327102 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:37.327114 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:37.328946 (+ 1832us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:37613: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":83,"threads_started":1}
W20260618 22:18:37.329298 9593 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:18:37.323823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:37961 (local address 127.0.98.126:37613)
0618 22:18:37.323989 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:37.323993 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:37.324118 (+ 125us) server_negotiation.cc:407] Connection header received
0618 22:18:37.324290 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:37.324293 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:37.324346 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:37.324432 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:37.324873 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:37.325401 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:37.326204 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:37.326437 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:37.326546 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:37.329117 (+ 2571us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.67:37961: BlockingRecv error: recv got EOF from 127.0.98.67:37961 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Jun 18 22:18:37 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:18:37 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:18:37.962939 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:37.954685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37613 (local address 127.0.98.65:45239)
0618 22:18:37.954974 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:37.954989 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:37.955083 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:37.955331 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:37.955333 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:37.955502 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:37.955594 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:37.955598 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:37.956288 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:37.956290 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:37.956850 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:37.956857 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:37.957322 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:37.957325 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:37.957388 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:37.960905 (+ 3517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:37.960918 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:37.962786 (+ 1868us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:37613: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":70,"threads_started":1}
W20260618 22:18:37.963109 9597 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:18:37.954744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:45239 (local address 127.0.98.126:37613)
0618 22:18:37.955026 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:37.955034 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:37.955047 (+ 13us) server_negotiation.cc:407] Connection header received
0618 22:18:37.955169 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:37.955171 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:37.955211 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:37.955284 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:37.955687 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:37.956199 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:37.956984 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:37.957208 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:37.957292 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:37.962943 (+ 5651us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.65:45239: BlockingRecv error: recv got EOF from 127.0.98.65:45239 (error 108)
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":112,"threads_started":1}
Jun 18 22:18:38 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:18:38 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:18:38.144812 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:38.139195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37613 (local address 127.0.98.66:52467)
0618 22:18:38.139510 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:38.139531 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:38.139681 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:38.139926 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:38.139928 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:38.140128 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:38.140234 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:38.140240 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:38.141021 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:38.141024 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:38.141606 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:38.141618 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:38.142118 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:38.142121 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:38.142192 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:38.142748 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:38.142760 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:38.144696 (+ 1936us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:37613: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":64,"threads_started":1}
W20260618 22:18:38.145004 9597 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:18:38.139334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:52467 (local address 127.0.98.126:37613)
0618 22:18:38.139509 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:38.139512 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:38.139606 (+ 94us) server_negotiation.cc:407] Connection header received
0618 22:18:38.139767 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:38.139769 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:38.139810 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:38.139882 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:38.140355 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:38.140902 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:38.141761 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:38.142001 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:38.142078 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:38.144858 (+ 2780us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.66:52467: BlockingRecv error: recv got EOF from 127.0.98.66:52467 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Jun 18 22:18:38 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:18:38 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:18:38.335642 9599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:38.329776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37613 (local address 127.0.98.67:35907)
0618 22:18:38.330056 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:38.330070 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:38.330169 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:38.330492 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:38.330495 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:38.330748 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:38.330875 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:38.330884 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:38.331757 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:38.331760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:38.332313 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:38.332321 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:38.332845 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:38.332848 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:38.332918 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:38.333517 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:38.333529 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:38.335455 (+ 1926us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:37613: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":97,"threads_started":1}
W20260618 22:18:38.335757 9597 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:18:38.329866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:35907 (local address 127.0.98.126:37613)
0618 22:18:38.330024 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:38.330027 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:38.330138 (+ 111us) server_negotiation.cc:407] Connection header received
0618 22:18:38.330284 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:38.330287 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:38.330343 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:38.330440 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:38.331003 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:38.331635 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:38.332412 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:38.332620 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:38.332695 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:38.335631 (+ 2936us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.67:35907: BlockingRecv error: recv got EOF from 127.0.98.67:35907 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Jun 18 22:18:38 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:18:38 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:18:38.969332 9600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:38.963703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37613 (local address 127.0.98.65:54431)
0618 22:18:38.964012 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:38.964028 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:38.964126 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:38.964375 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:38.964377 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:38.964550 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:38.964644 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:38.964649 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:38.965404 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:38.965406 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:38.966025 (+ 619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:38.966034 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:38.966543 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:38.966547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:38.966643 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:38.967231 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:38.967244 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:38.969171 (+ 1927us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:37613: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":113,"threads_started":1}
W20260618 22:18:38.969417 9601 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:18:38.963837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:54431 (local address 127.0.98.126:37613)
0618 22:18:38.964100 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:38.964104 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:38.964117 (+ 13us) server_negotiation.cc:407] Connection header received
0618 22:18:38.964208 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:38.964210 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:38.964256 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:38.964335 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:38.964762 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:38.965306 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:38.966128 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:38.966385 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:38.966531 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:38.969293 (+ 2762us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.65:54431: BlockingRecv error: recv got EOF from 127.0.98.65:54431 (error 108)
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":75,"threads_started":1}
Jun 18 22:18:39 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:18:39 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:18:39.151153 9602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:39.145637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37613 (local address 127.0.98.66:51909)
0618 22:18:39.145940 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:39.145953 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:39.146043 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:39.146271 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:39.146273 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:39.146447 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:39.146540 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:39.146545 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:39.147321 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:39.147323 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:39.147982 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:39.147991 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:39.148439 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:39.148442 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:39.148501 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:39.149003 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:39.149015 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:39.150985 (+ 1970us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:37613: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":88,"threads_started":1}
W20260618 22:18:39.151190 9601 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:18:39.145710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:51909 (local address 127.0.98.126:37613)
0618 22:18:39.145865 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:39.145870 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:39.146020 (+ 150us) server_negotiation.cc:407] Connection header received
0618 22:18:39.146122 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:39.146124 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:39.146166 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:39.146234 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:39.146632 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:39.147227 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:39.148121 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:39.148340 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:39.148422 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:39.151092 (+ 2670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.66:51909: BlockingRecv error: recv got EOF from 127.0.98.66:51909 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 18 22:18:39 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:18:39 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:18:39.342034 9603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:39.336443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37613 (local address 127.0.98.67:42355)
0618 22:18:39.336746 (+ 303us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:39.336760 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:39.336888 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:39.337166 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:39.337169 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:39.337374 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:39.337477 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:39.337483 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:39.338291 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:39.338296 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:39.338862 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:39.338870 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:39.339327 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:39.339331 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:39.339424 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:39.340012 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:39.340023 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:39.341825 (+ 1802us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:37613: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":108,"threads_started":1}
W20260618 22:18:39.342106 9601 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:18:39.336535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:42355 (local address 127.0.98.126:37613)
0618 22:18:39.336698 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:39.336702 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:39.336840 (+ 138us) server_negotiation.cc:407] Connection header received
0618 22:18:39.337007 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:39.337010 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:39.337061 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:39.337143 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:39.337595 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:39.338186 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:39.338983 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:39.339210 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:39.339314 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:39.342017 (+ 2703us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.67:42355: BlockingRecv error: recv got EOF from 127.0.98.67:42355 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Jun 18 22:18:39 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:18:39 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:18:39.975229 9604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:39.970077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37613 (local address 127.0.98.65:46115)
0618 22:18:39.970330 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:39.970344 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:39.970444 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:39.970672 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:39.970674 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:39.970841 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:39.970932 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:39.970936 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:39.971644 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:39.971646 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:39.972192 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:39.972200 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:39.972629 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:39.972631 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:39.972690 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:39.973189 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:39.973200 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:39.975121 (+ 1921us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:37613: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":66,"threads_started":1}
W20260618 22:18:39.975436 9605 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:18:39.970166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:46115 (local address 127.0.98.126:37613)
0618 22:18:39.970412 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:39.970416 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:39.970428 (+ 12us) server_negotiation.cc:407] Connection header received
0618 22:18:39.970529 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:39.970531 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:39.970570 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:39.970643 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:39.971034 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:39.971556 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:39.972305 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:39.972529 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:39.972616 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:39.975286 (+ 2670us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.65:46115: BlockingRecv error: recv got EOF from 127.0.98.65:46115 (error 108)
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":71,"threads_started":1}
Jun 18 22:18:40 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:18:40 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:18:40.157626 9606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:40.151937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37613 (local address 127.0.98.66:50483)
0618 22:18:40.152242 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:40.152256 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:40.152373 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:40.152786 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:40.152790 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:40.153054 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:40.153144 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:40.153149 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:40.154012 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:40.154017 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:40.154594 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:40.154601 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:40.155065 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:40.155069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:40.155154 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:40.155713 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:40.155725 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:40.157514 (+ 1789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:37613: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":76,"threads_started":1}
W20260618 22:18:40.157795 9605 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:18:40.152009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:50483 (local address 127.0.98.126:37613)
0618 22:18:40.152148 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:40.152152 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:40.152339 (+ 187us) server_negotiation.cc:407] Connection header received
0618 22:18:40.152545 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:40.152549 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:40.152602 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:40.152688 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:40.153276 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:40.153871 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:40.154715 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:40.154954 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:40.155063 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:40.157638 (+ 2575us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.66:50483: BlockingRecv error: recv got EOF from 127.0.98.66:50483 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 18 22:18:40 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:18:40 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:18:40.348693 9607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:40.342818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37613 (local address 127.0.98.67:39921)
0618 22:18:40.343122 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:40.343134 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:40.343264 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:40.343554 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:40.343557 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:40.343806 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:40.343893 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:40.343899 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:40.344651 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:40.344654 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:40.345246 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:40.345254 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:40.345788 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:40.345792 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:40.345878 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:40.346608 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:40.346633 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:40.348585 (+ 1952us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:37613: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":79,"threads_started":1}
W20260618 22:18:40.348822 9605 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:18:40.342914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:39921 (local address 127.0.98.126:37613)
0618 22:18:40.343042 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:40.343047 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:40.343225 (+ 178us) server_negotiation.cc:407] Connection header received
0618 22:18:40.343386 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:40.343390 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:40.343452 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:40.343536 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:40.344035 (+ 499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:40.344561 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:40.345398 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:40.345692 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:40.345786 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:40.348707 (+ 2921us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.67:39921: BlockingRecv error: recv got EOF from 127.0.98.67:39921 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 18 22:18:40 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:18:40 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:18:40.981110 9608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:40.976012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37613 (local address 127.0.98.65:59311)
0618 22:18:40.976284 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:40.976298 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:40.976382 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:40.976626 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:40.976628 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:40.976799 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:40.976889 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:40.976894 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:40.977586 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:40.977589 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:40.978166 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:40.978174 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:40.978589 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:40.978592 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:40.978661 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:40.979137 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:40.979148 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:40.980942 (+ 1794us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:37613: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":89,"threads_started":1}
W20260618 22:18:40.981181 9609 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:18:40.976057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:59311 (local address 127.0.98.126:37613)
0618 22:18:40.976320 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:40.976324 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:40.976340 (+ 16us) server_negotiation.cc:407] Connection header received
0618 22:18:40.976471 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:40.976473 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:40.976515 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:40.976591 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:40.976986 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:40.977499 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:40.978250 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:40.978472 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:40.978548 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:40.981062 (+ 2514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.65:59311: BlockingRecv error: recv got EOF from 127.0.98.65:59311 (error 108)
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":75,"threads_started":1}
Jun 18 22:18:41 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:18:41 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:18:41.163849 9610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:41.158365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37613 (local address 127.0.98.66:48441)
0618 22:18:41.158678 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:41.158705 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:41.158846 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:41.159142 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:41.159145 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:41.159352 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:41.159438 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:41.159443 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:41.160249 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:41.160254 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:41.160830 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:41.160838 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:41.161290 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:41.161294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:41.161378 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:41.161890 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:41.161933 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:41.163738 (+ 1805us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:37613: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":109,"threads_started":1}
W20260618 22:18:41.164059 9609 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:18:41.158504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:48441 (local address 127.0.98.126:37613)
0618 22:18:41.158668 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:41.158673 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:41.158792 (+ 119us) server_negotiation.cc:407] Connection header received
0618 22:18:41.158975 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:41.158978 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:41.159032 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:41.159123 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:41.159554 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:41.160114 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:41.160951 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:41.161173 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:41.161287 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:41.163939 (+ 2652us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.66:48441: BlockingRecv error: recv got EOF from 127.0.98.66:48441 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Jun 18 22:18:41 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:18:41 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:18:41.355700 9611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:41.349576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37613 (local address 127.0.98.67:35763)
0618 22:18:41.349937 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:41.349952 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:41.350049 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:41.350288 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:41.350290 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:41.350486 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:41.350583 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:41.350589 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:41.351378 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:41.351381 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:41.352042 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:41.352053 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:41.352550 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:41.352553 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:41.352631 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:41.353209 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:41.353221 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:41.355523 (+ 2302us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:37613: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":155,"threads_started":1}
W20260618 22:18:41.355871 9609 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:18:41.349682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:35763 (local address 127.0.98.126:37613)
0618 22:18:41.349843 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:41.349846 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:41.350021 (+ 175us) server_negotiation.cc:407] Connection header received
0618 22:18:41.350128 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:41.350130 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:41.350171 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:41.350243 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:41.350737 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:41.351260 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:41.352176 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:41.352401 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:41.352507 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:41.355704 (+ 3197us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.67:35763: BlockingRecv error: recv got EOF from 127.0.98.67:35763 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Jun 18 22:18:41 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:18:41 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:18:41.987648 9612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:41.982008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37613 (local address 127.0.98.65:60803)
0618 22:18:41.982274 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:41.982296 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:41.982398 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:41.982653 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:41.982655 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:41.982838 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:41.982931 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:41.982937 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:41.983670 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:41.983673 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:41.984225 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:41.984233 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:41.984703 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:41.984705 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:41.984773 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:41.985299 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:41.985309 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:41.987481 (+ 2172us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:37613: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":88,"threads_started":1}
W20260618 22:18:41.987807 9613 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:18:41.982077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:60803 (local address 127.0.98.126:37613)
0618 22:18:41.982315 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:41.982318 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:41.982365 (+ 47us) server_negotiation.cc:407] Connection header received
0618 22:18:41.982483 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:41.982485 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:41.982530 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:41.982614 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:41.983032 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:41.983569 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:41.984360 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:41.984597 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:41.984696 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:41.987635 (+ 2939us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.65:60803: BlockingRecv error: recv got EOF from 127.0.98.65:60803 (error 108)
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":100,"threads_started":1}
Jun 18 22:18:42 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:18:42 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:18:42.170225 9614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:42.164550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37613 (local address 127.0.98.66:37651)
0618 22:18:42.164919 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:42.164934 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:42.165068 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:42.165357 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:42.165360 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:42.165570 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:42.165657 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:42.165663 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:42.166486 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:42.166491 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:42.167069 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:42.167077 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:42.167547 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:42.167550 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:42.167636 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:42.168145 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:42.168156 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:42.170042 (+ 1886us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:37613: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":294,"thread_start_us":97,"threads_started":1}
W20260618 22:18:42.170266 9613 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:18:42.164721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:37651 (local address 127.0.98.126:37613)
0618 22:18:42.164886 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:42.164890 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:42.165017 (+ 127us) server_negotiation.cc:407] Connection header received
0618 22:18:42.165173 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:42.165177 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:42.165242 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:42.165332 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:42.165785 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:42.166374 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:42.167199 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:42.167426 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:42.167532 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:42.170164 (+ 2632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.66:37651: BlockingRecv error: recv got EOF from 127.0.98.66:37651 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Jun 18 22:18:42 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:18:42 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:18:42.362494 9615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:42.356783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37613 (local address 127.0.98.67:35863)
0618 22:18:42.357049 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:42.357062 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:42.357156 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:42.357372 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:42.357374 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:42.357596 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:42.357788 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:42.357797 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:42.358600 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:42.358610 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:42.359204 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:42.359213 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:42.359635 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:42.359637 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:42.359698 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:42.360217 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:42.360230 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:42.362361 (+ 2131us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:37613: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":101,"threads_started":1}
W20260618 22:18:42.362578 9613 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:18:42.356913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:35863 (local address 127.0.98.126:37613)
0618 22:18:42.357047 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:42.357051 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:42.357127 (+ 76us) server_negotiation.cc:407] Connection header received
0618 22:18:42.357229 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:42.357231 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:42.357275 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:42.357340 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:42.357934 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:42.358493 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:42.359319 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:42.359538 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:42.359617 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:42.362486 (+ 2869us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.67:35863: BlockingRecv error: recv got EOF from 127.0.98.67:35863 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Jun 18 22:18:42 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:18:42 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:18:42.998432 9616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:42.988467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37613 (local address 127.0.98.65:33185)
0618 22:18:42.988780 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:42.988796 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:42.988925 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:42.989232 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:42.989235 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:42.989456 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:42.989551 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:42.989556 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:42.990346 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:42.990349 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:42.990922 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:42.990929 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:42.991388 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:42.991391 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:42.991457 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:42.996142 (+ 4685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:42.996156 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:42.998227 (+ 2071us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:37613: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":107,"threads_started":1}
W20260618 22:18:42.998493 9617 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:18:42.988564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:33185 (local address 127.0.98.126:37613)
0618 22:18:42.988882 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:42.988887 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:42.988902 (+ 15us) server_negotiation.cc:407] Connection header received
0618 22:18:42.989045 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:42.989048 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:42.989097 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:42.989190 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:42.989660 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:42.990228 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:42.991033 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:42.991271 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:42.991351 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:42.998361 (+ 7010us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.65:33185: BlockingRecv error: recv got EOF from 127.0.98.65:33185 (error 108)
Metrics: {"server-negotiator.queue_time_us":232,"thread_start_us":66,"threads_started":1}
Jun 18 22:18:43 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:18:43 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:18:43.176700 9618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:43.171133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37613 (local address 127.0.98.66:45711)
0618 22:18:43.171388 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:43.171403 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:43.171538 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:43.171809 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:43.171812 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:43.172016 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:43.172101 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:43.172106 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:43.172838 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:43.172843 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:43.173400 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:43.173406 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:43.173856 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:43.173860 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:43.173981 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:43.174457 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:43.174468 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:43.176537 (+ 2069us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:37613: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":110,"threads_started":1}
W20260618 22:18:43.176743 9617 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:18:43.171208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:45711 (local address 127.0.98.126:37613)
0618 22:18:43.171346 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:43.171350 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:43.171489 (+ 139us) server_negotiation.cc:407] Connection header received
0618 22:18:43.171640 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:43.171644 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:43.171692 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:43.171775 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:43.172215 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:43.172732 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:43.173512 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:43.173745 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:43.173854 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:43.176649 (+ 2795us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.66:45711: BlockingRecv error: recv got EOF from 127.0.98.66:45711 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Jun 18 22:18:43 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:18:43 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.67@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:18:43.369583 9619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:43.363280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37613 (local address 127.0.98.67:51071)
0618 22:18:43.363622 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:43.363646 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:43.363831 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:43.364122 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:43.364126 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:43.364367 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:43.364460 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:43.364466 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:43.365312 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:43.365324 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:43.366035 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:43.366045 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:43.366538 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:43.366543 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:43.366637 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:43.367252 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:43.367272 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:43.369400 (+ 2128us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:37613: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":89,"threads_started":1}
W20260618 22:18:43.369619 9617 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:18:43.363410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:51071 (local address 127.0.98.126:37613)
0618 22:18:43.363585 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:43.363591 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:43.363765 (+ 174us) server_negotiation.cc:407] Connection header received
0618 22:18:43.363946 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:43.363951 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:43.364006 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:43.364100 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:43.364592 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:43.365166 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:43.366168 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:43.366416 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:43.366538 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:43.369515 (+ 2977us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.67:51071: BlockingRecv error: recv got EOF from 127.0.98.67:51071 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Jun 18 22:18:43 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:18:43 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.65@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:18:44.004560 9620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:43.999352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37613 (local address 127.0.98.65:42893)
0618 22:18:43.999660 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:43.999674 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:43.999765 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:44.000004 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:44.000006 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:44.000180 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:44.000267 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:44.000272 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:44.001025 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:44.001027 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:44.001575 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:44.001582 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:44.002024 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:44.002028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:44.002116 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:44.002664 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:44.002675 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:44.004372 (+ 1697us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:37613: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":84,"threads_started":1}
W20260618 22:18:44.004623 9621 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:18:43.999463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:42893 (local address 127.0.98.126:37613)
0618 22:18:43.999693 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:43.999696 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:43.999727 (+ 31us) server_negotiation.cc:407] Connection header received
0618 22:18:43.999851 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:43.999853 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:43.999892 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:43.999966 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:44.000388 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:44.000932 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:44.001671 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:44.001889 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:44.002019 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:44.004496 (+ 2477us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.65:42893: BlockingRecv error: recv got EOF from 127.0.98.65:42893 (error 108)
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":54,"threads_started":1}
Jun 18 22:18:44 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
Jun 18 22:18:44 dist-test-slave-g6r4 krb5kdc[8563](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.66@KRBTEST.COM for oryx/127.0.98.126@KRBTEST.COM, Server not found in Kerberos database
I20260618 22:18:44.183081 9622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:44.177736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37613 (local address 127.0.98.66:47701)
0618 22:18:44.178099 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:44.178112 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:44.178203 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:44.178422 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:44.178424 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:44.178596 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:44.178697 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:44.178702 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:44.179430 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:44.179432 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:44.179971 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:44.179978 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:44.180384 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:44.180386 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:44.180469 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:44.180948 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:44.180959 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:44.182902 (+ 1943us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.126:37613: Server oryx/127.0.98.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":78,"threads_started":1}
W20260618 22:18:44.183116 9621 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:18:44.177801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:47701 (local address 127.0.98.126:37613)
0618 22:18:44.177954 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:44.177959 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:44.178175 (+ 216us) server_negotiation.cc:407] Connection header received
0618 22:18:44.178271 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:44.178273 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:44.178312 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:44.178378 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:44.178813 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:44.179333 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:44.180068 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:44.180291 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:44.180367 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:44.183014 (+ 2647us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.66:47701: BlockingRecv error: recv got EOF from 127.0.98.66:47701 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
I20260618 22:18:44.256603 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 9146
I20260618 22:18:44.261786 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 9298
I20260618 22:18:44.267397 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 9430
I20260618 22:18:44.272325 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 9076
2026-06-18T22:18:44Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17971 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-18T22:18:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:18:44Z Disabled control of system clock
I20260618 22:18:44.302824 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821065380026-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821065380026-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821065380026-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:35817
--webserver_interface=127.0.98.126
--webserver_port=0
--builtin_ntp_servers=127.0.98.84:37571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:35817
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260618 22:18:44.408900 9626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:18:44.409179 9626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:18:44.409214 9626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:18:44.413111 9626 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:18:44.413166 9626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:18:44.413182 9626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:18:44.413198 9626 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:18:44.413213 9626 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:18:44.417990 9626 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:37571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821065380026-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821065380026-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:35817
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:35817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821065380026-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:18:44.418985 9626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:18:44.420274 9626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:18:44.429096 9631 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:44.429132 9634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:44.429188 9632 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:44.429262 9626 server_base.cc:1034] running on GCE node
I20260618 22:18:44.429966 9626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:18:44.431131 9626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:18:44.432364 9626 hybrid_clock.cc:648] HybridClock initialized: now 1781821124432355 us; error 35 us; skew 500 ppm
I20260618 22:18:44.434239 9626 webserver.cc:460] Webserver started at http://127.0.98.126:45903/ using document root <none> and password file <none>
I20260618 22:18:44.434835 9626 fs_manager.cc:359] Metadata directory not provided
I20260618 22:18:44.434928 9626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:18:44.435153 9626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:18:44.437151 9626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821065380026-393-0/minicluster-data/master-0/data/instance:
uuid: "404d4b6783de44fa8ce93883730e2807"
format_stamp: "Formatted at 2026-06-18 22:18:44 on dist-test-slave-g6r4"
server_key: "dbce4adb0624f45a78a1d972f75d6882"
server_key_iv: "d3a61a37451ef9ca89b258a02c7c0b3b"
server_key_version: "encryptionkey@0"
I20260618 22:18:44.437745 9626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821065380026-393-0/minicluster-data/master-0/wal/instance:
uuid: "404d4b6783de44fa8ce93883730e2807"
format_stamp: "Formatted at 2026-06-18 22:18:44 on dist-test-slave-g6r4"
server_key: "dbce4adb0624f45a78a1d972f75d6882"
server_key_iv: "d3a61a37451ef9ca89b258a02c7c0b3b"
server_key_version: "encryptionkey@0"
I20260618 22:18:44.441452 9626 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260618 22:18:44.443933 9640 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:44.445163 9626 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:18:44.445314 9626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821065380026-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821065380026-393-0/minicluster-data/master-0/wal
uuid: "404d4b6783de44fa8ce93883730e2807"
format_stamp: "Formatted at 2026-06-18 22:18:44 on dist-test-slave-g6r4"
server_key: "dbce4adb0624f45a78a1d972f75d6882"
server_key_iv: "d3a61a37451ef9ca89b258a02c7c0b3b"
server_key_version: "encryptionkey@0"
I20260618 22:18:44.445425 9626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821065380026-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821065380026-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821065380026-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:18:44.459420 9626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:18:44.460135 9626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:18:44.460335 9626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:18:44.467164 9626 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:35817
I20260618 22:18:44.467195 9692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:35817 every 8 connection(s)
I20260618 22:18:44.468051 9626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
I20260618 22:18:44.471051 9693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:18:44.476061 9693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 404d4b6783de44fa8ce93883730e2807: Bootstrap starting.
I20260618 22:18:44.478296 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 9626
I20260618 22:18:44.478380 9693 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 404d4b6783de44fa8ce93883730e2807: Neither blocks nor log segments found. Creating new log.
I20260618 22:18:44.478428 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821065380026-393-0/minicluster-data/master-0/wal/instance
I20260618 22:18:44.478754 393 external_mini_cluster.cc:1428] Setting key f1e460f12c0ede70528bf358dd7742a8
I20260618 22:18:44.479239 9693 log.cc:824] T 00000000000000000000000000000000 P 404d4b6783de44fa8ce93883730e2807: Log is configured to *not* fsync() on all Append() calls
I20260618 22:18:44.481237 9693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 404d4b6783de44fa8ce93883730e2807: No bootstrap required, opened a new log
I20260618 22:18:44.485292 9693 raft_consensus.cc:348] T 00000000000000000000000000000000 P 404d4b6783de44fa8ce93883730e2807 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "404d4b6783de44fa8ce93883730e2807" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 35817 } }
I20260618 22:18:44.485553 9693 raft_consensus.cc:374] T 00000000000000000000000000000000 P 404d4b6783de44fa8ce93883730e2807 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:18:44.485603 9693 raft_consensus.cc:729] T 00000000000000000000000000000000 P 404d4b6783de44fa8ce93883730e2807 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 404d4b6783de44fa8ce93883730e2807, State: Initialized, Role: FOLLOWER
I20260618 22:18:44.486114 9693 consensus_queue.cc:260] T 00000000000000000000000000000000 P 404d4b6783de44fa8ce93883730e2807 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "404d4b6783de44fa8ce93883730e2807" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 35817 } }
I20260618 22:18:44.486277 9693 raft_consensus.cc:388] T 00000000000000000000000000000000 P 404d4b6783de44fa8ce93883730e2807 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:18:44.486352 9693 raft_consensus.cc:482] T 00000000000000000000000000000000 P 404d4b6783de44fa8ce93883730e2807 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:18:44.486447 9693 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 404d4b6783de44fa8ce93883730e2807 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:18:44.487555 9693 raft_consensus.cc:504] T 00000000000000000000000000000000 P 404d4b6783de44fa8ce93883730e2807 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "404d4b6783de44fa8ce93883730e2807" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 35817 } }
I20260618 22:18:44.487887 9697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:44.479908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55624 (local address 127.0.98.126:35817)
0618 22:18:44.480495 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:44.480510 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:44.480542 (+ 32us) server_negotiation.cc:407] Connection header received
0618 22:18:44.481138 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:44.481155 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:44.481523 (+ 368us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:44.481790 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:44.482191 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:44.483133 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:44.484358 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:44.484809 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:44.484932 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:44.485793 (+ 861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:44.485799 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:44.485815 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:18:44.486327 (+ 512us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:44.486404 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:18:44.486430 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:44.486531 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:44.486700 (+ 169us) server_negotiation.cc:299] Negotiation successful
0618 22:18:44.486953 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":131,"threads_started":1}
I20260618 22:18:44.487900 9693 leader_election.cc:302] T 00000000000000000000000000000000 P 404d4b6783de44fa8ce93883730e2807 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 404d4b6783de44fa8ce93883730e2807; no voters:
I20260618 22:18:44.488224 9693 leader_election.cc:288] T 00000000000000000000000000000000 P 404d4b6783de44fa8ce93883730e2807 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:18:44.488349 9698 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 404d4b6783de44fa8ce93883730e2807 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:18:44.488579 9698 raft_consensus.cc:686] T 00000000000000000000000000000000 P 404d4b6783de44fa8ce93883730e2807 [term 1 LEADER]: Becoming Leader. State: Replica: 404d4b6783de44fa8ce93883730e2807, State: Running, Role: LEADER
I20260618 22:18:44.489344 9693 sys_catalog.cc:564] T 00000000000000000000000000000000 P 404d4b6783de44fa8ce93883730e2807 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:18:44.489513 9698 consensus_queue.cc:237] T 00000000000000000000000000000000 P 404d4b6783de44fa8ce93883730e2807 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "404d4b6783de44fa8ce93883730e2807" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 35817 } }
I20260618 22:18:44.491173 9699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 404d4b6783de44fa8ce93883730e2807 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "404d4b6783de44fa8ce93883730e2807" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "404d4b6783de44fa8ce93883730e2807" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 35817 } } }
I20260618 22:18:44.491300 9699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 404d4b6783de44fa8ce93883730e2807 [sys.catalog]: This master's current role is: LEADER
I20260618 22:18:44.491377 9700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 404d4b6783de44fa8ce93883730e2807 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 404d4b6783de44fa8ce93883730e2807. Latest consensus state: current_term: 1 leader_uuid: "404d4b6783de44fa8ce93883730e2807" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "404d4b6783de44fa8ce93883730e2807" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 35817 } } }
I20260618 22:18:44.491480 9700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 404d4b6783de44fa8ce93883730e2807 [sys.catalog]: This master's current role is: LEADER
I20260618 22:18:44.491802 9712 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:18:44.494361 9712 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:18:44.499146 9712 catalog_manager.cc:1347] Generated new cluster ID: 1fbe917dab68487da954fd509cade7eb
I20260618 22:18:44.499213 9712 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:18:44.508162 9712 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:18:44.509263 9712 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:18:44.518271 9712 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 404d4b6783de44fa8ce93883730e2807: Generated new TSK 0
I20260618 22:18:44.519034 9712 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:18:44.523734 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821065380026-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821065380026-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821065380026-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:35817
--builtin_ntp_servers=127.0.98.84:37571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260618 22:18:44.632826 9717 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:18:44.633104 9717 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:18:44.633173 9717 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:18:44.637161 9717 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:18:44.637247 9717 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:18:44.637343 9717 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260618 22:18:44.642027 9717 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:37571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821065380026-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821065380026-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:35817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9717
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821065380026-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:18:44.643105 9717 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:18:44.644461 9717 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:18:44.653906 9722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:44.653959 9725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:44.653923 9723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:44.654222 9717 server_base.cc:1034] running on GCE node
I20260618 22:18:44.654652 9717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:18:44.655308 9717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:18:44.656487 9717 hybrid_clock.cc:648] HybridClock initialized: now 1781821124656472 us; error 26 us; skew 500 ppm
I20260618 22:18:44.658291 9717 webserver.cc:460] Webserver started at http://127.0.98.65:36823/ using document root <none> and password file <none>
I20260618 22:18:44.658885 9717 fs_manager.cc:359] Metadata directory not provided
I20260618 22:18:44.658978 9717 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:18:44.659219 9717 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:18:44.661269 9717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821065380026-393-0/minicluster-data/ts-0/data/instance:
uuid: "a5b7fdd6ba1f43e0953a842f21aa7f52"
format_stamp: "Formatted at 2026-06-18 22:18:44 on dist-test-slave-g6r4"
server_key: "4d45e87b64118c8145bd5a606ea1f0eb"
server_key_iv: "8f9a6e234fbd7b22f6c334cfec6fc9b6"
server_key_version: "encryptionkey@0"
I20260618 22:18:44.661840 9717 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821065380026-393-0/minicluster-data/ts-0/wal/instance:
uuid: "a5b7fdd6ba1f43e0953a842f21aa7f52"
format_stamp: "Formatted at 2026-06-18 22:18:44 on dist-test-slave-g6r4"
server_key: "4d45e87b64118c8145bd5a606ea1f0eb"
server_key_iv: "8f9a6e234fbd7b22f6c334cfec6fc9b6"
server_key_version: "encryptionkey@0"
I20260618 22:18:44.665467 9717 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:18:44.667914 9731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:44.669107 9717 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:18:44.669235 9717 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821065380026-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821065380026-393-0/minicluster-data/ts-0/wal
uuid: "a5b7fdd6ba1f43e0953a842f21aa7f52"
format_stamp: "Formatted at 2026-06-18 22:18:44 on dist-test-slave-g6r4"
server_key: "4d45e87b64118c8145bd5a606ea1f0eb"
server_key_iv: "8f9a6e234fbd7b22f6c334cfec6fc9b6"
server_key_version: "encryptionkey@0"
I20260618 22:18:44.669342 9717 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821065380026-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821065380026-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821065380026-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:18:44.694792 9717 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:18:44.695520 9717 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:18:44.695722 9717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:18:44.696308 9717 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:18:44.697256 9717 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:18:44.697329 9717 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:44.697407 9717 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:18:44.697449 9717 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:44.706038 9717 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:45409
I20260618 22:18:44.706099 9844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:45409 every 8 connection(s)
I20260618 22:18:44.706949 9717 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
I20260618 22:18:44.710273 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 9717
I20260618 22:18:44.710395 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821065380026-393-0/minicluster-data/ts-0/wal/instance
I20260618 22:18:44.710711 393 external_mini_cluster.cc:1428] Setting key 676fc2514e3ba6ab6f97704a448bdac1
I20260618 22:18:44.712852 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821065380026-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821065380026-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821065380026-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:35817
--builtin_ntp_servers=127.0.98.84:37571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260618 22:18:44.717003 9697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:44.708657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:32851 (local address 127.0.98.126:35817)
0618 22:18:44.708785 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:44.708788 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:44.709719 (+ 931us) server_negotiation.cc:407] Connection header received
0618 22:18:44.711541 (+ 1822us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:44.711544 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:44.711589 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:44.711674 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:44.712779 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:44.713384 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:44.714785 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:44.715208 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:44.715330 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:44.716377 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:44.716383 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:44.716386 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:18:44.716410 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:44.716437 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:18:44.716449 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:44.716538 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:44.716721 (+ 183us) server_negotiation.cc:299] Negotiation successful
0618 22:18:44.716792 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":48}
I20260618 22:18:44.717746 9847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:44.708990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:35817 (local address 127.0.98.65:32851)
0618 22:18:44.709592 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:44.709631 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:44.711285 (+ 1654us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:44.711876 (+ 591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:44.711885 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:44.711985 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:44.712621 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:44.712643 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:44.713525 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:44.713532 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:44.714671 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:44.714679 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:44.715352 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:44.715354 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:44.715492 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:44.716123 (+ 631us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 22:18:44.716124 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:44.716162 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 22:18:44.716219 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:18:44.716567 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:18:44.716575 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:18:44.716579 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 22:18:44.716693 (+ 114us) client_negotiation.cc:240] Negotiation successful
0618 22:18:44.716931 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":112,"threads_started":1}
I20260618 22:18:44.719026 9845 heartbeater.cc:347] Connected to a master server at 127.0.98.126:35817
I20260618 22:18:44.719249 9845 heartbeater.cc:464] Registering TS with master...
I20260618 22:18:44.719704 9845 heartbeater.cc:511] Master 127.0.98.126:35817 requested a full tablet report, sending...
I20260618 22:18:44.721263 9657 ts_manager.cc:194] Registered new tserver with Master: a5b7fdd6ba1f43e0953a842f21aa7f52 (127.0.98.65:45409)
I20260618 22:18:44.722975 9657 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.65:32851
W20260618 22:18:44.819741 9848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:18:44.819983 9848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:18:44.820020 9848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:18:44.823980 9848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:18:44.824033 9848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:18:44.824084 9848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260618 22:18:44.828819 9848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:37571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821065380026-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821065380026-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:35817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821065380026-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:18:44.829767 9848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:18:44.831104 9848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:18:44.840516 9853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:44.840525 9856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:44.840533 9854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:44.840574 9848 server_base.cc:1034] running on GCE node
I20260618 22:18:44.841235 9848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:18:44.841938 9848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:18:44.843113 9848 hybrid_clock.cc:648] HybridClock initialized: now 1781821124843092 us; error 33 us; skew 500 ppm
I20260618 22:18:44.844930 9848 webserver.cc:460] Webserver started at http://127.0.98.66:34939/ using document root <none> and password file <none>
I20260618 22:18:44.845487 9848 fs_manager.cc:359] Metadata directory not provided
I20260618 22:18:44.845547 9848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:18:44.845773 9848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:18:44.847754 9848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821065380026-393-0/minicluster-data/ts-1/data/instance:
uuid: "2ecd01c3d7aa4a7ab5d8338cbb03b36c"
format_stamp: "Formatted at 2026-06-18 22:18:44 on dist-test-slave-g6r4"
server_key: "16248d8032da304a403b17db676aedcb"
server_key_iv: "e537583e89937e4ae85c635ca67bd799"
server_key_version: "encryptionkey@0"
I20260618 22:18:44.848327 9848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821065380026-393-0/minicluster-data/ts-1/wal/instance:
uuid: "2ecd01c3d7aa4a7ab5d8338cbb03b36c"
format_stamp: "Formatted at 2026-06-18 22:18:44 on dist-test-slave-g6r4"
server_key: "16248d8032da304a403b17db676aedcb"
server_key_iv: "e537583e89937e4ae85c635ca67bd799"
server_key_version: "encryptionkey@0"
I20260618 22:18:44.852037 9848 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260618 22:18:44.854449 9862 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:44.855674 9848 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:18:44.855786 9848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821065380026-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821065380026-393-0/minicluster-data/ts-1/wal
uuid: "2ecd01c3d7aa4a7ab5d8338cbb03b36c"
format_stamp: "Formatted at 2026-06-18 22:18:44 on dist-test-slave-g6r4"
server_key: "16248d8032da304a403b17db676aedcb"
server_key_iv: "e537583e89937e4ae85c635ca67bd799"
server_key_version: "encryptionkey@0"
I20260618 22:18:44.855890 9848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821065380026-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821065380026-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821065380026-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:18:44.901659 9848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:18:44.902551 9848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:18:44.902776 9848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:18:44.903591 9848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:18:44.904673 9848 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:18:44.904744 9848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:44.904811 9848 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:18:44.904860 9848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:44.913793 9848 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:45095
I20260618 22:18:44.913815 9975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:45095 every 8 connection(s)
I20260618 22:18:44.914722 9848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
I20260618 22:18:44.919389 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 9848
I20260618 22:18:44.919509 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821065380026-393-0/minicluster-data/ts-1/wal/instance
I20260618 22:18:44.919781 393 external_mini_cluster.cc:1428] Setting key 3c0ea7aa18f01a606a113df14d40c7e1
I20260618 22:18:44.921634 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821065380026-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821065380026-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821065380026-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:35817
--builtin_ntp_servers=127.0.98.84:37571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260618 22:18:44.924225 9697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:44.916462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:45147 (local address 127.0.98.126:35817)
0618 22:18:44.916599 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:44.916602 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:44.917743 (+ 1141us) server_negotiation.cc:407] Connection header received
0618 22:18:44.919225 (+ 1482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:44.919228 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:44.919284 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:44.919360 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:44.920325 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:44.921053 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:44.922340 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:44.922567 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:44.922677 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:44.923730 (+ 1053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:44.923733 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:44.923736 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:18:44.923748 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:44.923775 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:18:44.923789 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:44.923863 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:44.924029 (+ 166us) server_negotiation.cc:299] Negotiation successful
0618 22:18:44.924092 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260618 22:18:44.925024 9978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:44.916912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:35817 (local address 127.0.98.66:45147)
0618 22:18:44.917590 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:44.917635 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:44.919007 (+ 1372us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:44.919554 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:44.919563 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:44.919681 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:44.920184 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:44.920196 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:44.921174 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:44.921184 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:44.922188 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:44.922198 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:44.922708 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:44.922712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:44.922878 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:44.923474 (+ 596us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 22:18:44.923476 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:44.923515 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 22:18:44.923571 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:18:44.923880 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:18:44.923895 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:18:44.923900 (+ 5us) client_negotiation.cc:769] Sending connection context
0618 22:18:44.924017 (+ 117us) client_negotiation.cc:240] Negotiation successful
0618 22:18:44.924309 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":108,"threads_started":1}
I20260618 22:18:44.926218 9976 heartbeater.cc:347] Connected to a master server at 127.0.98.126:35817
I20260618 22:18:44.926465 9976 heartbeater.cc:464] Registering TS with master...
I20260618 22:18:44.927047 9976 heartbeater.cc:511] Master 127.0.98.126:35817 requested a full tablet report, sending...
I20260618 22:18:44.928179 9657 ts_manager.cc:194] Registered new tserver with Master: 2ecd01c3d7aa4a7ab5d8338cbb03b36c (127.0.98.66:45095)
I20260618 22:18:44.928727 9657 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.66:45147
W20260618 22:18:45.029412 9979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:18:45.029635 9979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:18:45.029670 9979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:18:45.033685 9979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:18:45.033736 9979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:18:45.033834 9979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260618 22:18:45.038440 9979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:37571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821065380026-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821065380026-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:35817
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821065380026-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:18:45.039721 9979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:18:45.041492 9979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:18:45.050432 9985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:45.050451 9984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:45.050451 9987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:45.050722 9979 server_base.cc:1034] running on GCE node
I20260618 22:18:45.051044 9979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:18:45.051625 9979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:18:45.052837 9979 hybrid_clock.cc:648] HybridClock initialized: now 1781821125052796 us; error 51 us; skew 500 ppm
I20260618 22:18:45.054764 9979 webserver.cc:460] Webserver started at http://127.0.98.67:45163/ using document root <none> and password file <none>
I20260618 22:18:45.055318 9979 fs_manager.cc:359] Metadata directory not provided
I20260618 22:18:45.055378 9979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:18:45.055619 9979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:18:45.057570 9979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821065380026-393-0/minicluster-data/ts-2/data/instance:
uuid: "bc37800d25f542eb84118db90f4f0411"
format_stamp: "Formatted at 2026-06-18 22:18:45 on dist-test-slave-g6r4"
server_key: "5d91e6f81293068a54f24ee50936128f"
server_key_iv: "c85dd6d4266ffe694d1e50057f57fe37"
server_key_version: "encryptionkey@0"
I20260618 22:18:45.058133 9979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821065380026-393-0/minicluster-data/ts-2/wal/instance:
uuid: "bc37800d25f542eb84118db90f4f0411"
format_stamp: "Formatted at 2026-06-18 22:18:45 on dist-test-slave-g6r4"
server_key: "5d91e6f81293068a54f24ee50936128f"
server_key_iv: "c85dd6d4266ffe694d1e50057f57fe37"
server_key_version: "encryptionkey@0"
I20260618 22:18:45.061740 9979 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260618 22:18:45.064327 9993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:45.065500 9979 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:18:45.065637 9979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821065380026-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821065380026-393-0/minicluster-data/ts-2/wal
uuid: "bc37800d25f542eb84118db90f4f0411"
format_stamp: "Formatted at 2026-06-18 22:18:45 on dist-test-slave-g6r4"
server_key: "5d91e6f81293068a54f24ee50936128f"
server_key_iv: "c85dd6d4266ffe694d1e50057f57fe37"
server_key_version: "encryptionkey@0"
I20260618 22:18:45.065747 9979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821065380026-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821065380026-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821065380026-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:18:45.086344 9979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:18:45.087031 9979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:18:45.087224 9979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:18:45.087778 9979 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:18:45.088670 9979 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:18:45.088739 9979 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:45.088835 9979 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:18:45.088887 9979 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:45.097859 9979 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:45089
I20260618 22:18:45.097965 10106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:45089 every 8 connection(s)
I20260618 22:18:45.098740 9979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
I20260618 22:18:45.107203 9697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:45.100474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:53167 (local address 127.0.98.126:35817)
0618 22:18:45.100635 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:45.100640 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:45.101621 (+ 981us) server_negotiation.cc:407] Connection header received
0618 22:18:45.103008 (+ 1387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:45.103012 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:45.103058 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:45.103132 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:45.103961 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:45.104472 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:45.105567 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:45.105773 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:45.105847 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:45.106832 (+ 985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:45.106835 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:45.106837 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:18:45.106848 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:45.106866 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:18:45.106873 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:45.106930 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:45.107051 (+ 121us) server_negotiation.cc:299] Negotiation successful
0618 22:18:45.107095 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260618 22:18:45.107995 10109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:45.100868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:35817 (local address 127.0.98.67:53167)
0618 22:18:45.101476 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:45.101515 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:45.102819 (+ 1304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:45.103279 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:45.103285 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:45.103364 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:45.103837 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:45.103849 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:45.104578 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:45.104581 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:45.105460 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:45.105468 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:45.105889 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:45.105892 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:45.106077 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:45.106590 (+ 513us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 22:18:45.106591 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:45.106628 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 22:18:45.106684 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:18:45.106940 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:18:45.106945 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:18:45.106949 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 22:18:45.107036 (+ 87us) client_negotiation.cc:240] Negotiation successful
0618 22:18:45.107235 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":102,"threads_started":1}
I20260618 22:18:45.108064 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 9979
I20260618 22:18:45.108168 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781821065380026-393-0/minicluster-data/ts-2/wal/instance
I20260618 22:18:45.108429 393 external_mini_cluster.cc:1428] Setting key 77bbccd238b92ca07ed864cf231c38a5
I20260618 22:18:45.109103 10107 heartbeater.cc:347] Connected to a master server at 127.0.98.126:35817
I20260618 22:18:45.109315 10107 heartbeater.cc:464] Registering TS with master...
I20260618 22:18:45.109771 10107 heartbeater.cc:511] Master 127.0.98.126:35817 requested a full tablet report, sending...
I20260618 22:18:45.110698 9657 ts_manager.cc:194] Registered new tserver with Master: bc37800d25f542eb84118db90f4f0411 (127.0.98.67:45089)
I20260618 22:18:45.111229 9657 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.67:53167
I20260618 22:18:45.121923 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260618 22:18:45.124604 9697 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:18:45.123780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55638 (local address 127.0.98.126:35817)
0618 22:18:45.123904 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:45.123908 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:45.124092 (+ 184us) server_negotiation.cc:407] Connection header received
0618 22:18:45.124177 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:45.124179 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:45.124213 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:45.124291 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:45.124467 (+ 176us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55638: BlockingRecv error: recv got EOF from 127.0.0.1:55638 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20260618 22:18:45.125082 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 9717
I20260618 22:18:45.130512 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 9848
I20260618 22:18:45.135454 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 9979
I20260618 22:18:45.140434 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 9626
2026-06-18T22:18:45Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (865 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-18T22:18:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:18:45Z Disabled control of system clock
I20260618 22:18:45.168654 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821065380026-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821065380026-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821065380026-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:37591
--webserver_interface=127.0.98.126
--webserver_port=0
--builtin_ntp_servers=127.0.98.84:34173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:37591
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260618 22:18:45.275354 10118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:18:45.275592 10118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:18:45.275627 10118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:18:45.279480 10118 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:18:45.279533 10118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:18:45.279548 10118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:18:45.279565 10118 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:18:45.279579 10118 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:18:45.284278 10118 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:34173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821065380026-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821065380026-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:37591
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:37591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10118
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821065380026-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:18:45.285225 10118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:18:45.286541 10118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:18:45.294806 10126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:45.294855 10124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:45.294814 10123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:45.295150 10118 server_base.cc:1034] running on GCE node
I20260618 22:18:45.295670 10118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:18:45.296761 10118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:18:45.297983 10118 hybrid_clock.cc:648] HybridClock initialized: now 1781821125297963 us; error 34 us; skew 500 ppm
I20260618 22:18:45.299736 10118 webserver.cc:460] Webserver started at http://127.0.98.126:36103/ using document root <none> and password file <none>
I20260618 22:18:45.300290 10118 fs_manager.cc:359] Metadata directory not provided
I20260618 22:18:45.300381 10118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:18:45.300621 10118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:18:45.302651 10118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821065380026-393-0/minicluster-data/master-0/data/instance:
uuid: "f6805d2ca07c4d8783ec11140ca1047e"
format_stamp: "Formatted at 2026-06-18 22:18:45 on dist-test-slave-g6r4"
server_key: "9d2ed7e6c27c94821a1984cffc1c7cb7"
server_key_iv: "7349240b44eeedbad9ba23abe4cbd32b"
server_key_version: "encryptionkey@0"
I20260618 22:18:45.303202 10118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821065380026-393-0/minicluster-data/master-0/wal/instance:
uuid: "f6805d2ca07c4d8783ec11140ca1047e"
format_stamp: "Formatted at 2026-06-18 22:18:45 on dist-test-slave-g6r4"
server_key: "9d2ed7e6c27c94821a1984cffc1c7cb7"
server_key_iv: "7349240b44eeedbad9ba23abe4cbd32b"
server_key_version: "encryptionkey@0"
I20260618 22:18:45.306702 10118 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:18:45.308982 10132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:45.310256 10118 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:18:45.310385 10118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821065380026-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821065380026-393-0/minicluster-data/master-0/wal
uuid: "f6805d2ca07c4d8783ec11140ca1047e"
format_stamp: "Formatted at 2026-06-18 22:18:45 on dist-test-slave-g6r4"
server_key: "9d2ed7e6c27c94821a1984cffc1c7cb7"
server_key_iv: "7349240b44eeedbad9ba23abe4cbd32b"
server_key_version: "encryptionkey@0"
I20260618 22:18:45.310503 10118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821065380026-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821065380026-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821065380026-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:18:45.314850 10118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:18:45.315423 10118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:18:45.315599 10118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:18:45.322190 10118 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:37591
I20260618 22:18:45.322206 10184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:37591 every 8 connection(s)
I20260618 22:18:45.323040 10118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
I20260618 22:18:45.323901 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 10118
I20260618 22:18:45.324011 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821065380026-393-0/minicluster-data/master-0/wal/instance
I20260618 22:18:45.324297 393 external_mini_cluster.cc:1428] Setting key b704fdcce856bea83033aee5d636569d
I20260618 22:18:45.326260 10185 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:18:45.329955 10188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:45.325557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47912 (local address 127.0.98.126:37591)
0618 22:18:45.326620 (+ 1063us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:45.326635 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:45.326668 (+ 33us) server_negotiation.cc:407] Connection header received
0618 22:18:45.327272 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:45.327286 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:45.327605 (+ 319us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:45.327871 (+ 266us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:45.328695 (+ 824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:45.328700 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:45.328713 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:18:45.328722 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:45.328781 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:18:45.328802 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:45.328923 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:45.329022 (+ 99us) server_negotiation.cc:299] Negotiation successful
0618 22:18:45.329277 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":727,"thread_start_us":606,"threads_started":1}
I20260618 22:18:45.332428 10185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6805d2ca07c4d8783ec11140ca1047e: Bootstrap starting.
I20260618 22:18:45.334477 10185 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6805d2ca07c4d8783ec11140ca1047e: Neither blocks nor log segments found. Creating new log.
I20260618 22:18:45.335156 10185 log.cc:824] T 00000000000000000000000000000000 P f6805d2ca07c4d8783ec11140ca1047e: Log is configured to *not* fsync() on all Append() calls
I20260618 22:18:45.337010 10185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6805d2ca07c4d8783ec11140ca1047e: No bootstrap required, opened a new log
I20260618 22:18:45.340152 10185 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6805d2ca07c4d8783ec11140ca1047e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6805d2ca07c4d8783ec11140ca1047e" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 37591 } }
I20260618 22:18:45.340384 10185 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6805d2ca07c4d8783ec11140ca1047e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:18:45.340487 10185 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6805d2ca07c4d8783ec11140ca1047e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6805d2ca07c4d8783ec11140ca1047e, State: Initialized, Role: FOLLOWER
I20260618 22:18:45.340917 10185 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6805d2ca07c4d8783ec11140ca1047e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6805d2ca07c4d8783ec11140ca1047e" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 37591 } }
I20260618 22:18:45.341063 10185 raft_consensus.cc:388] T 00000000000000000000000000000000 P f6805d2ca07c4d8783ec11140ca1047e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:18:45.341156 10185 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6805d2ca07c4d8783ec11140ca1047e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:18:45.341254 10185 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6805d2ca07c4d8783ec11140ca1047e [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:18:45.342098 10185 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6805d2ca07c4d8783ec11140ca1047e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6805d2ca07c4d8783ec11140ca1047e" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 37591 } }
I20260618 22:18:45.342434 10185 leader_election.cc:302] T 00000000000000000000000000000000 P f6805d2ca07c4d8783ec11140ca1047e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6805d2ca07c4d8783ec11140ca1047e; no voters:
I20260618 22:18:45.342770 10185 leader_election.cc:288] T 00000000000000000000000000000000 P f6805d2ca07c4d8783ec11140ca1047e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:18:45.342885 10190 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6805d2ca07c4d8783ec11140ca1047e [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:18:45.343139 10190 raft_consensus.cc:686] T 00000000000000000000000000000000 P f6805d2ca07c4d8783ec11140ca1047e [term 1 LEADER]: Becoming Leader. State: Replica: f6805d2ca07c4d8783ec11140ca1047e, State: Running, Role: LEADER
I20260618 22:18:45.343519 10190 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6805d2ca07c4d8783ec11140ca1047e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6805d2ca07c4d8783ec11140ca1047e" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 37591 } }
I20260618 22:18:45.343930 10185 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6805d2ca07c4d8783ec11140ca1047e [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:18:45.345192 10192 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6805d2ca07c4d8783ec11140ca1047e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6805d2ca07c4d8783ec11140ca1047e. Latest consensus state: current_term: 1 leader_uuid: "f6805d2ca07c4d8783ec11140ca1047e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6805d2ca07c4d8783ec11140ca1047e" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 37591 } } }
I20260618 22:18:45.345238 10191 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6805d2ca07c4d8783ec11140ca1047e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f6805d2ca07c4d8783ec11140ca1047e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6805d2ca07c4d8783ec11140ca1047e" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 37591 } } }
I20260618 22:18:45.345304 10192 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6805d2ca07c4d8783ec11140ca1047e [sys.catalog]: This master's current role is: LEADER
I20260618 22:18:45.345330 10191 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6805d2ca07c4d8783ec11140ca1047e [sys.catalog]: This master's current role is: LEADER
I20260618 22:18:45.345623 10199 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:18:45.348285 10199 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:18:45.353216 10199 catalog_manager.cc:1347] Generated new cluster ID: 1e288395bafe4b75a72e64acb28bb62d
I20260618 22:18:45.353286 10199 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:18:45.360365 10199 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:18:45.360649 10199 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:18:45.364578 10199 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f6805d2ca07c4d8783ec11140ca1047e: Generated new TSK 0
I20260618 22:18:45.365217 10199 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:18:45.367656 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821065380026-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821065380026-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821065380026-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:37591
--builtin_ntp_servers=127.0.98.84:34173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260618 22:18:45.476631 10209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:18:45.476866 10209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:18:45.476903 10209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:18:45.480914 10209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:18:45.480969 10209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:18:45.481024 10209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260618 22:18:45.485675 10209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:34173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821065380026-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821065380026-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:37591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821065380026-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:18:45.486708 10209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:18:45.487993 10209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:18:45.497094 10215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:45.497140 10214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:45.497094 10217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:45.497340 10209 server_base.cc:1034] running on GCE node
I20260618 22:18:45.497784 10209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:18:45.498436 10209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:18:45.499624 10209 hybrid_clock.cc:648] HybridClock initialized: now 1781821125499589 us; error 48 us; skew 500 ppm
I20260618 22:18:45.501313 10209 webserver.cc:460] Webserver started at http://127.0.98.65:40377/ using document root <none> and password file <none>
I20260618 22:18:45.501856 10209 fs_manager.cc:359] Metadata directory not provided
I20260618 22:18:45.501986 10209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:18:45.502194 10209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:18:45.504194 10209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821065380026-393-0/minicluster-data/ts-0/data/instance:
uuid: "8bfd7f388c1f42c48469eb56b38b09fd"
format_stamp: "Formatted at 2026-06-18 22:18:45 on dist-test-slave-g6r4"
server_key: "3be457c09067b92437120f8f5335f6fc"
server_key_iv: "fd2702bd7053a5743e8e49e09b08b0a3"
server_key_version: "encryptionkey@0"
I20260618 22:18:45.504732 10209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821065380026-393-0/minicluster-data/ts-0/wal/instance:
uuid: "8bfd7f388c1f42c48469eb56b38b09fd"
format_stamp: "Formatted at 2026-06-18 22:18:45 on dist-test-slave-g6r4"
server_key: "3be457c09067b92437120f8f5335f6fc"
server_key_iv: "fd2702bd7053a5743e8e49e09b08b0a3"
server_key_version: "encryptionkey@0"
I20260618 22:18:45.508235 10209 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260618 22:18:45.510578 10223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:45.511685 10209 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260618 22:18:45.511855 10209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821065380026-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821065380026-393-0/minicluster-data/ts-0/wal
uuid: "8bfd7f388c1f42c48469eb56b38b09fd"
format_stamp: "Formatted at 2026-06-18 22:18:45 on dist-test-slave-g6r4"
server_key: "3be457c09067b92437120f8f5335f6fc"
server_key_iv: "fd2702bd7053a5743e8e49e09b08b0a3"
server_key_version: "encryptionkey@0"
I20260618 22:18:45.511968 10209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821065380026-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821065380026-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821065380026-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:18:45.516535 10209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:18:45.517094 10209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:18:45.517271 10209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:18:45.517783 10209 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:18:45.518666 10209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:18:45.518734 10209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:45.518811 10209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:18:45.518867 10209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:45.527732 10209 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:33265
I20260618 22:18:45.527765 10336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:33265 every 8 connection(s)
I20260618 22:18:45.528553 10209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
I20260618 22:18:45.533728 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 10209
I20260618 22:18:45.533861 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821065380026-393-0/minicluster-data/ts-0/wal/instance
I20260618 22:18:45.534148 393 external_mini_cluster.cc:1428] Setting key 11ce7deaba4d930e1d3825a5791fdcd6
I20260618 22:18:45.534492 10188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:45.530324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:59981 (local address 127.0.98.126:37591)
0618 22:18:45.530463 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:45.530467 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:45.531503 (+ 1036us) server_negotiation.cc:407] Connection header received
0618 22:18:45.532685 (+ 1182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:45.532688 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:45.532722 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:45.532806 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:45.534023 (+ 1217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:45.534027 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:45.534029 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:18:45.534039 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:45.534057 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:18:45.534065 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:45.534166 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:45.534324 (+ 158us) server_negotiation.cc:299] Negotiation successful
0618 22:18:45.534367 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260618 22:18:45.535256 10339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:45.530724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37591 (local address 127.0.98.65:59981)
0618 22:18:45.531353 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:45.531392 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:45.532500 (+ 1108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:45.532946 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:45.532952 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:45.533032 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:45.533647 (+ 615us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 22:18:45.533648 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:45.533688 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 22:18:45.533772 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:18:45.534154 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:18:45.534163 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:18:45.534168 (+ 5us) client_negotiation.cc:769] Sending connection context
0618 22:18:45.534339 (+ 171us) client_negotiation.cc:240] Negotiation successful
0618 22:18:45.534551 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":107,"threads_started":1}
I20260618 22:18:45.536343 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821065380026-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821065380026-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821065380026-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:37591
--builtin_ntp_servers=127.0.98.84:34173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260618 22:18:45.536437 10337 heartbeater.cc:347] Connected to a master server at 127.0.98.126:37591
I20260618 22:18:45.536651 10337 heartbeater.cc:464] Registering TS with master...
I20260618 22:18:45.537062 10337 heartbeater.cc:511] Master 127.0.98.126:37591 requested a full tablet report, sending...
I20260618 22:18:45.538511 10149 ts_manager.cc:194] Registered new tserver with Master: 8bfd7f388c1f42c48469eb56b38b09fd (127.0.98.65:33265)
W20260618 22:18:45.642506 10340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:18:45.642735 10340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:18:45.642778 10340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:18:45.646937 10340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:18:45.646994 10340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:18:45.647048 10340 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260618 22:18:45.651623 10340 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:34173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821065380026-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821065380026-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:37591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10340
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821065380026-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:18:45.652631 10340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:18:45.653980 10340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:18:45.663270 10348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:45.663244 10345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:45.663295 10346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:45.663421 10340 server_base.cc:1034] running on GCE node
I20260618 22:18:45.663899 10340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:18:45.664645 10340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:18:45.665823 10340 hybrid_clock.cc:648] HybridClock initialized: now 1781821125665804 us; error 33 us; skew 500 ppm
I20260618 22:18:45.667527 10340 webserver.cc:460] Webserver started at http://127.0.98.66:33517/ using document root <none> and password file <none>
I20260618 22:18:45.668023 10340 fs_manager.cc:359] Metadata directory not provided
I20260618 22:18:45.668080 10340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:18:45.668228 10340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:18:45.670264 10340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821065380026-393-0/minicluster-data/ts-1/data/instance:
uuid: "4563204148b94ab6958e13c514c575bc"
format_stamp: "Formatted at 2026-06-18 22:18:45 on dist-test-slave-g6r4"
server_key: "b6ac9bb20587b8a8cdae45f3df137e6b"
server_key_iv: "d2b7f97a55d760fadda8edadb601bf4e"
server_key_version: "encryptionkey@0"
I20260618 22:18:45.670780 10340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821065380026-393-0/minicluster-data/ts-1/wal/instance:
uuid: "4563204148b94ab6958e13c514c575bc"
format_stamp: "Formatted at 2026-06-18 22:18:45 on dist-test-slave-g6r4"
server_key: "b6ac9bb20587b8a8cdae45f3df137e6b"
server_key_iv: "d2b7f97a55d760fadda8edadb601bf4e"
server_key_version: "encryptionkey@0"
I20260618 22:18:45.674183 10340 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:18:45.676473 10354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:45.677690 10340 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:18:45.677824 10340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821065380026-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821065380026-393-0/minicluster-data/ts-1/wal
uuid: "4563204148b94ab6958e13c514c575bc"
format_stamp: "Formatted at 2026-06-18 22:18:45 on dist-test-slave-g6r4"
server_key: "b6ac9bb20587b8a8cdae45f3df137e6b"
server_key_iv: "d2b7f97a55d760fadda8edadb601bf4e"
server_key_version: "encryptionkey@0"
I20260618 22:18:45.677987 10340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821065380026-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821065380026-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821065380026-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:18:45.682547 10340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:18:45.683293 10340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:18:45.683475 10340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:18:45.684100 10340 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:18:45.684962 10340 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:18:45.685031 10340 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:45.685096 10340 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:18:45.685127 10340 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:45.693511 10340 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:45303
I20260618 22:18:45.693562 10467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:45303 every 8 connection(s)
I20260618 22:18:45.694432 10340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
I20260618 22:18:45.700112 10188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:45.696091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:42765 (local address 127.0.98.126:37591)
0618 22:18:45.696226 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:45.696230 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:45.697269 (+ 1039us) server_negotiation.cc:407] Connection header received
0618 22:18:45.698565 (+ 1296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:45.698567 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:45.698602 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:45.698669 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:45.699708 (+ 1039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:45.699711 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:45.699713 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:18:45.699723 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:45.699741 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:18:45.699748 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:45.699825 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:45.699965 (+ 140us) server_negotiation.cc:299] Negotiation successful
0618 22:18:45.700004 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260618 22:18:45.700937 10470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:45.696472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37591 (local address 127.0.98.66:42765)
0618 22:18:45.697129 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:45.697168 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:45.698360 (+ 1192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:45.698816 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:45.698822 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:45.698922 (+ 100us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:45.699474 (+ 552us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 22:18:45.699476 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:45.699512 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 22:18:45.699574 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:18:45.699826 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:18:45.699831 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:18:45.699834 (+ 3us) client_negotiation.cc:769] Sending connection context
0618 22:18:45.699970 (+ 136us) client_negotiation.cc:240] Negotiation successful
0618 22:18:45.700144 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":105,"threads_started":1}
I20260618 22:18:45.702114 10468 heartbeater.cc:347] Connected to a master server at 127.0.98.126:37591
I20260618 22:18:45.702350 10468 heartbeater.cc:464] Registering TS with master...
I20260618 22:18:45.702420 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 10340
I20260618 22:18:45.702533 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821065380026-393-0/minicluster-data/ts-1/wal/instance
I20260618 22:18:45.702811 10468 heartbeater.cc:511] Master 127.0.98.126:37591 requested a full tablet report, sending...
I20260618 22:18:45.702844 393 external_mini_cluster.cc:1428] Setting key 9c86b1982fad9282e7846fd9f5395441
I20260618 22:18:45.703821 10148 ts_manager.cc:194] Registered new tserver with Master: 4563204148b94ab6958e13c514c575bc (127.0.98.66:45303)
I20260618 22:18:45.704795 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821065380026-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821065380026-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821065380026-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:37591
--builtin_ntp_servers=127.0.98.84:34173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260618 22:18:45.811012 10471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:18:45.811232 10471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:18:45.811269 10471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:18:45.815740 10471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:18:45.815801 10471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:18:45.815894 10471 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260618 22:18:45.820842 10471 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:34173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821065380026-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821065380026-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:37591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10471
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821065380026-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:18:45.821779 10471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:18:45.823089 10471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:18:45.832212 10476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:45.832294 10479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:45.832283 10477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:45.832651 10471 server_base.cc:1034] running on GCE node
I20260618 22:18:45.833014 10471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:18:45.833563 10471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:18:45.834769 10471 hybrid_clock.cc:648] HybridClock initialized: now 1781821125834756 us; error 26 us; skew 500 ppm
I20260618 22:18:45.836606 10471 webserver.cc:460] Webserver started at http://127.0.98.67:32791/ using document root <none> and password file <none>
I20260618 22:18:45.837188 10471 fs_manager.cc:359] Metadata directory not provided
I20260618 22:18:45.837255 10471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:18:45.837486 10471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:18:45.839564 10471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821065380026-393-0/minicluster-data/ts-2/data/instance:
uuid: "a4fd10863a2940cd9c8b6f69f9439a97"
format_stamp: "Formatted at 2026-06-18 22:18:45 on dist-test-slave-g6r4"
server_key: "69c30f8e720b9b01a3ccae78918a13e6"
server_key_iv: "fa5644f4edf4230972cecfb4a1b7a99f"
server_key_version: "encryptionkey@0"
I20260618 22:18:45.840116 10471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821065380026-393-0/minicluster-data/ts-2/wal/instance:
uuid: "a4fd10863a2940cd9c8b6f69f9439a97"
format_stamp: "Formatted at 2026-06-18 22:18:45 on dist-test-slave-g6r4"
server_key: "69c30f8e720b9b01a3ccae78918a13e6"
server_key_iv: "fa5644f4edf4230972cecfb4a1b7a99f"
server_key_version: "encryptionkey@0"
I20260618 22:18:45.843628 10471 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260618 22:18:45.846129 10485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:45.847438 10471 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260618 22:18:45.847564 10471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821065380026-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821065380026-393-0/minicluster-data/ts-2/wal
uuid: "a4fd10863a2940cd9c8b6f69f9439a97"
format_stamp: "Formatted at 2026-06-18 22:18:45 on dist-test-slave-g6r4"
server_key: "69c30f8e720b9b01a3ccae78918a13e6"
server_key_iv: "fa5644f4edf4230972cecfb4a1b7a99f"
server_key_version: "encryptionkey@0"
I20260618 22:18:45.847671 10471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821065380026-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821065380026-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821065380026-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:18:45.852488 10471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:18:45.853065 10471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:18:45.853253 10471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:18:45.853785 10471 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:18:45.854707 10471 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:18:45.854756 10471 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:45.854827 10471 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:18:45.854867 10471 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:45.864012 10471 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:38729
I20260618 22:18:45.864046 10598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:38729 every 8 connection(s)
I20260618 22:18:45.865033 10471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
I20260618 22:18:45.870523 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 10471
I20260618 22:18:45.870656 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781821065380026-393-0/minicluster-data/ts-2/wal/instance
I20260618 22:18:45.871012 10188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:45.866762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:38719 (local address 127.0.98.126:37591)
0618 22:18:45.866890 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:45.866894 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:45.867965 (+ 1071us) server_negotiation.cc:407] Connection header received
0618 22:18:45.869189 (+ 1224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:45.869192 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:45.869228 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:45.869305 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:45.870416 (+ 1111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:45.870421 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:45.870424 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:18:45.870439 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:45.870464 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:18:45.870477 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:45.870598 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:45.870802 (+ 204us) server_negotiation.cc:299] Negotiation successful
0618 22:18:45.870850 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260618 22:18:45.871053 393 external_mini_cluster.cc:1428] Setting key 43e925a45821b12b89e68452bba039cc
I20260618 22:18:45.871915 10601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:45.867168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:37591 (local address 127.0.98.67:38719)
0618 22:18:45.867830 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:45.867867 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:45.868973 (+ 1106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:45.869430 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:45.869436 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:45.869517 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:45.870146 (+ 629us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 22:18:45.870147 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:45.870184 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 22:18:45.870264 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:18:45.870573 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:18:45.870577 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:18:45.870581 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 22:18:45.870776 (+ 195us) client_negotiation.cc:240] Negotiation successful
0618 22:18:45.870973 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":109,"threads_started":1}
I20260618 22:18:45.873040 10599 heartbeater.cc:347] Connected to a master server at 127.0.98.126:37591
I20260618 22:18:45.873253 10599 heartbeater.cc:464] Registering TS with master...
I20260618 22:18:45.873638 10599 heartbeater.cc:511] Master 127.0.98.126:37591 requested a full tablet report, sending...
I20260618 22:18:45.874608 10149 ts_manager.cc:194] Registered new tserver with Master: a4fd10863a2940cd9c8b6f69f9439a97 (127.0.98.67:38729)
I20260618 22:18:45.885003 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260618 22:18:45.887648 10188 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:18:45.886864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47928 (local address 127.0.98.126:37591)
0618 22:18:45.887033 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:45.887036 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:45.887101 (+ 65us) server_negotiation.cc:407] Connection header received
0618 22:18:45.887191 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:45.887193 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:45.887221 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:45.887284 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:45.887539 (+ 255us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47928: BlockingRecv error: recv got EOF from 127.0.0.1:47928 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260618 22:18:45.887899 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 10209
I20260618 22:18:45.893615 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 10340
I20260618 22:18:45.898756 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 10471
I20260618 22:18:45.903657 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 10118
2026-06-18T22:18:45Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (763 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:18:45 dist-test-slave-g6r4 krb5kdc[10609](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:18:45 dist-test-slave-g6r4 krb5kdc[10609](info): set up 2 sockets
Jun 18 22:18:45 dist-test-slave-g6r4 krb5kdc[10609](info): commencing operation
krb5kdc: starting...
W20260618 22:18:46.951346 393 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/krb5kdc/test-user.keytab.
Jun 18 22:18:47 dist-test-slave-g6r4 krb5kdc[10609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:18:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:18:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260618 22:18:47.072197 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:41859
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:39035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:41859
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:18:47.176751 10622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:18:47.176999 10622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:18:47.177037 10622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:18:47.181011 10622 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:18:47.181062 10622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:18:47.181082 10622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:18:47.181099 10622 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:18:47.181114 10622 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:18:47.186131 10622 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:39035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:41859
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:41859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10622
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:18:47.187053 10622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:18:47.187987 10622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:18:47.196508 10627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:47.196542 10622 server_base.cc:1034] running on GCE node
W20260618 22:18:47.196578 10630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:47.196508 10628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:47.197387 10622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:18:47.198438 10622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:18:47.199620 10622 hybrid_clock.cc:648] HybridClock initialized: now 1781821127199594 us; error 37 us; skew 500 ppm
Jun 18 22:18:47 dist-test-slave-g6r4 krb5kdc[10609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821127, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:18:47.202184 10622 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260618 22:18:47.202986 10622 webserver.cc:460] Webserver started at http://127.0.98.126:41937/ using document root <none> and password file <none>
I20260618 22:18:47.203516 10622 fs_manager.cc:359] Metadata directory not provided
I20260618 22:18:47.203576 10622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:18:47.203797 10622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:18:47.205726 10622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/minicluster-data/master-0/data/instance:
uuid: "641152b53bf24f16b1e900c782a05a38"
format_stamp: "Formatted at 2026-06-18 22:18:47 on dist-test-slave-g6r4"
server_key: "369eed76876240e0b6f146b59531a0da"
server_key_iv: "64a916969faf1dd1bffca9740aa33595"
server_key_version: "encryptionkey@0"
I20260618 22:18:47.206329 10622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/minicluster-data/master-0/wal/instance:
uuid: "641152b53bf24f16b1e900c782a05a38"
format_stamp: "Formatted at 2026-06-18 22:18:47 on dist-test-slave-g6r4"
server_key: "369eed76876240e0b6f146b59531a0da"
server_key_iv: "64a916969faf1dd1bffca9740aa33595"
server_key_version: "encryptionkey@0"
I20260618 22:18:47.209806 10622 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260618 22:18:47.212044 10637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:47.213145 10622 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:18:47.213264 10622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/minicluster-data/master-0/wal
uuid: "641152b53bf24f16b1e900c782a05a38"
format_stamp: "Formatted at 2026-06-18 22:18:47 on dist-test-slave-g6r4"
server_key: "369eed76876240e0b6f146b59531a0da"
server_key_iv: "64a916969faf1dd1bffca9740aa33595"
server_key_version: "encryptionkey@0"
I20260618 22:18:47.213358 10622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:18:47.224234 10622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:18:47.226827 10622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:18:47.227018 10622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:18:47.233814 10622 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:41859
I20260618 22:18:47.233945 10689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:41859 every 8 connection(s)
I20260618 22:18:47.234766 10622 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
I20260618 22:18:47.237375 10690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:18:47.237658 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 10622
I20260618 22:18:47.237788 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/minicluster-data/master-0/wal/instance
I20260618 22:18:47.238113 393 external_mini_cluster.cc:1428] Setting key 1cb4c75cad486aca9cdb6c9fbf1b8af0
I20260618 22:18:47.244354 10690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 641152b53bf24f16b1e900c782a05a38: Bootstrap starting.
I20260618 22:18:47.246913 10690 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 641152b53bf24f16b1e900c782a05a38: Neither blocks nor log segments found. Creating new log.
Jun 18 22:18:47 dist-test-slave-g6r4 krb5kdc[10609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821127, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:18:47.247809 10690 log.cc:824] T 00000000000000000000000000000000 P 641152b53bf24f16b1e900c782a05a38: Log is configured to *not* fsync() on all Append() calls
I20260618 22:18:47.249612 10690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 641152b53bf24f16b1e900c782a05a38: No bootstrap required, opened a new log
I20260618 22:18:47.252607 10693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:47.239399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53964 (local address 127.0.98.126:41859)
0618 22:18:47.239809 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:47.239828 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:47.239868 (+ 40us) server_negotiation.cc:407] Connection header received
0618 22:18:47.240734 (+ 866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:47.240758 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:47.241162 (+ 404us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:47.241528 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:47.242250 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:47.243707 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:47.244589 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:47.245237 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:47.245387 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:47.248041 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:47.248046 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:47.248059 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:47.248085 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:47.249856 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:47.250398 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:47.250410 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:47.250415 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:47.250520 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:47.250809 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:47.250810 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:47.250812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:47.251170 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:47.251325 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:47.251627 (+ 302us) server_negotiation.cc:299] Negotiation successful
0618 22:18:47.251891 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":114,"threads_started":1}
I20260618 22:18:47.253619 10690 raft_consensus.cc:348] T 00000000000000000000000000000000 P 641152b53bf24f16b1e900c782a05a38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "641152b53bf24f16b1e900c782a05a38" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41859 } }
I20260618 22:18:47.253885 10690 raft_consensus.cc:374] T 00000000000000000000000000000000 P 641152b53bf24f16b1e900c782a05a38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:18:47.253968 10690 raft_consensus.cc:729] T 00000000000000000000000000000000 P 641152b53bf24f16b1e900c782a05a38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 641152b53bf24f16b1e900c782a05a38, State: Initialized, Role: FOLLOWER
I20260618 22:18:47.254398 10690 consensus_queue.cc:260] T 00000000000000000000000000000000 P 641152b53bf24f16b1e900c782a05a38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "641152b53bf24f16b1e900c782a05a38" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41859 } }
I20260618 22:18:47.254549 10690 raft_consensus.cc:388] T 00000000000000000000000000000000 P 641152b53bf24f16b1e900c782a05a38 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:18:47.254647 10690 raft_consensus.cc:482] T 00000000000000000000000000000000 P 641152b53bf24f16b1e900c782a05a38 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:18:47.254776 10690 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 641152b53bf24f16b1e900c782a05a38 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:18:47.255614 10690 raft_consensus.cc:504] T 00000000000000000000000000000000 P 641152b53bf24f16b1e900c782a05a38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "641152b53bf24f16b1e900c782a05a38" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41859 } }
I20260618 22:18:47.255959 10690 leader_election.cc:302] T 00000000000000000000000000000000 P 641152b53bf24f16b1e900c782a05a38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 641152b53bf24f16b1e900c782a05a38; no voters:
I20260618 22:18:47.256254 10690 leader_election.cc:288] T 00000000000000000000000000000000 P 641152b53bf24f16b1e900c782a05a38 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:18:47.256349 10695 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 641152b53bf24f16b1e900c782a05a38 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:18:47.256600 10695 raft_consensus.cc:686] T 00000000000000000000000000000000 P 641152b53bf24f16b1e900c782a05a38 [term 1 LEADER]: Becoming Leader. State: Replica: 641152b53bf24f16b1e900c782a05a38, State: Running, Role: LEADER
I20260618 22:18:47.256933 10695 consensus_queue.cc:237] T 00000000000000000000000000000000 P 641152b53bf24f16b1e900c782a05a38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "641152b53bf24f16b1e900c782a05a38" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41859 } }
I20260618 22:18:47.257478 10690 sys_catalog.cc:564] T 00000000000000000000000000000000 P 641152b53bf24f16b1e900c782a05a38 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:18:47.258687 10697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 641152b53bf24f16b1e900c782a05a38 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 641152b53bf24f16b1e900c782a05a38. Latest consensus state: current_term: 1 leader_uuid: "641152b53bf24f16b1e900c782a05a38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "641152b53bf24f16b1e900c782a05a38" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41859 } } }
I20260618 22:18:47.258807 10697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 641152b53bf24f16b1e900c782a05a38 [sys.catalog]: This master's current role is: LEADER
I20260618 22:18:47.259126 10696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 641152b53bf24f16b1e900c782a05a38 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "641152b53bf24f16b1e900c782a05a38" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "641152b53bf24f16b1e900c782a05a38" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 41859 } } }
I20260618 22:18:47.259287 10696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 641152b53bf24f16b1e900c782a05a38 [sys.catalog]: This master's current role is: LEADER
W20260618 22:18:47.262274 10711 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 641152b53bf24f16b1e900c782a05a38: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 22:18:47.262336 10711 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 22:18:47.262418 10704 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:18:47.263051 10704 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:18:47.267907 10704 catalog_manager.cc:1347] Generated new cluster ID: 4b5c149c43674904810392b8a354a20e
I20260618 22:18:47.267970 10704 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:18:47.287322 10704 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:18:47.288254 10704 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:18:47.293139 10704 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 641152b53bf24f16b1e900c782a05a38: Generated new TSK 0
I20260618 22:18:47.293825 10704 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab.
I20260618 22:18:47.325423 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:41859
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:39035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:18:47.433739 10716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:18:47.434088 10716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:18:47.434178 10716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:18:47.438122 10716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:18:47.438207 10716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:18:47.438294 10716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260618 22:18:47.443526 10716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:39035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:41859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:18:47.444581 10716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:18:47.445580 10716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:18:47.454746 10724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:47.454746 10721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:47.454762 10722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:47.455271 10716 server_base.cc:1034] running on GCE node
I20260618 22:18:47.455644 10716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:18:47.456180 10716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:18:47.457384 10716 hybrid_clock.cc:648] HybridClock initialized: now 1781821127457367 us; error 28 us; skew 500 ppm
Jun 18 22:18:47 dist-test-slave-g6r4 krb5kdc[10609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821127, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:18:47.460276 10716 init.cc:377] Logged in from keytab as kudu/127.0.98.65@KRBTEST.COM (short username kudu)
I20260618 22:18:47.461129 10716 webserver.cc:460] Webserver started at http://127.0.98.65:35329/ using document root <none> and password file <none>
I20260618 22:18:47.461673 10716 fs_manager.cc:359] Metadata directory not provided
I20260618 22:18:47.461760 10716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:18:47.462045 10716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:18:47.463960 10716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/minicluster-data/ts-0/data/instance:
uuid: "3f37594f859841eab5768680ec166be4"
format_stamp: "Formatted at 2026-06-18 22:18:47 on dist-test-slave-g6r4"
server_key: "9990ccee1630af86cb96a94f2892e01a"
server_key_iv: "045d595750085d3695bf9cd14d029d57"
server_key_version: "encryptionkey@0"
I20260618 22:18:47.464512 10716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/minicluster-data/ts-0/wal/instance:
uuid: "3f37594f859841eab5768680ec166be4"
format_stamp: "Formatted at 2026-06-18 22:18:47 on dist-test-slave-g6r4"
server_key: "9990ccee1630af86cb96a94f2892e01a"
server_key_iv: "045d595750085d3695bf9cd14d029d57"
server_key_version: "encryptionkey@0"
I20260618 22:18:47.468056 10716 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260618 22:18:47.470374 10731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:47.471608 10716 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:18:47.471741 10716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/minicluster-data/ts-0/wal
uuid: "3f37594f859841eab5768680ec166be4"
format_stamp: "Formatted at 2026-06-18 22:18:47 on dist-test-slave-g6r4"
server_key: "9990ccee1630af86cb96a94f2892e01a"
server_key_iv: "045d595750085d3695bf9cd14d029d57"
server_key_version: "encryptionkey@0"
I20260618 22:18:47.471848 10716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:18:47.498039 10716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:18:47.500852 10716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:18:47.501055 10716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:18:47.501643 10716 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:18:47.502449 10716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:18:47.502521 10716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:47.502588 10716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:18:47.502624 10716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:47.511181 10716 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:43243
I20260618 22:18:47.511205 10844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:43243 every 8 connection(s)
I20260618 22:18:47.512008 10716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
Jun 18 22:18:47 dist-test-slave-g6r4 krb5kdc[10609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821127, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:18:47.521749 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 10716
I20260618 22:18:47.521878 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/minicluster-data/ts-0/wal/instance
I20260618 22:18:47.522248 393 external_mini_cluster.cc:1428] Setting key b3bae6c43c1a85ace1bc836502b8ca30
I20260618 22:18:47.524897 10693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:47.513658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:40695 (local address 127.0.98.126:41859)
0618 22:18:47.513785 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:47.513789 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:47.514842 (+ 1053us) server_negotiation.cc:407] Connection header received
0618 22:18:47.515618 (+ 776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:47.515620 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:47.515663 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:47.515732 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:47.516894 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:47.517449 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:47.518592 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:47.518805 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:47.518889 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:47.521233 (+ 2344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:47.521237 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:47.521239 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:47.521272 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:47.522953 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:47.523429 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:47.523434 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:47.523436 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:47.523520 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:47.523844 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:47.523849 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:47.523851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:47.524033 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:47.524134 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:47.524610 (+ 476us) server_negotiation.cc:299] Negotiation successful
0618 22:18:47.524725 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260618 22:18:47.525928 10847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:47.514078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:41859 (local address 127.0.98.65:40695)
0618 22:18:47.514686 (+ 608us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:47.514727 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:47.515452 (+ 725us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:47.515861 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:47.515867 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:47.516264 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:47.516731 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:47.516744 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:47.517551 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:47.517554 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:47.518489 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:47.518498 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:47.518943 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:47.518946 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:47.519079 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:47.519662 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:47.519677 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:47.521085 (+ 1408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:18:47.523053 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:47.523059 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:47.523072 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:47.523324 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:47.523611 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:47.523615 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:47.523617 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:47.523745 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:47.524142 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:18:47.524150 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:18:47.524382 (+ 232us) client_negotiation.cc:769] Sending connection context
0618 22:18:47.524557 (+ 175us) client_negotiation.cc:240] Negotiation successful
0618 22:18:47.524859 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":99,"threads_started":1}
I20260618 22:18:47.527114 10845 heartbeater.cc:347] Connected to a master server at 127.0.98.126:41859
I20260618 22:18:47.527351 10845 heartbeater.cc:464] Registering TS with master...
I20260618 22:18:47.527913 10845 heartbeater.cc:511] Master 127.0.98.126:41859 requested a full tablet report, sending...
I20260618 22:18:47.529412 10654 ts_manager.cc:194] Registered new tserver with Master: 3f37594f859841eab5768680ec166be4 (127.0.98.65:43243)
I20260618 22:18:47.530543 10654 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.65@KRBTEST.COM'} at 127.0.98.65:40695
WARNING: no policy specified for kudu/127.0.98.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab.
I20260618 22:18:47.549434 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:41859
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:39035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:18:47.656081 10850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:18:47.656325 10850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:18:47.656368 10850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:18:47.660554 10850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:18:47.660615 10850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:18:47.660672 10850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260618 22:18:47.665740 10850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:39035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:41859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:18:47.666731 10850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:18:47.667646 10850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:18:47.677058 10855 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:47.677037 10856 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:47.677094 10858 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:47.677412 10850 server_base.cc:1034] running on GCE node
I20260618 22:18:47.677804 10850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:18:47.678485 10850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:18:47.679670 10850 hybrid_clock.cc:648] HybridClock initialized: now 1781821127679658 us; error 67 us; skew 500 ppm
Jun 18 22:18:47 dist-test-slave-g6r4 krb5kdc[10609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821127, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:18:47.682462 10850 init.cc:377] Logged in from keytab as kudu/127.0.98.66@KRBTEST.COM (short username kudu)
I20260618 22:18:47.683210 10850 webserver.cc:460] Webserver started at http://127.0.98.66:44461/ using document root <none> and password file <none>
I20260618 22:18:47.683701 10850 fs_manager.cc:359] Metadata directory not provided
I20260618 22:18:47.683756 10850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:18:47.683918 10850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:18:47.686362 10850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/minicluster-data/ts-1/data/instance:
uuid: "a49d50a39627487cbbc80e1f42fab1ef"
format_stamp: "Formatted at 2026-06-18 22:18:47 on dist-test-slave-g6r4"
server_key: "a700e19d628529d7ed72c71499701721"
server_key_iv: "d1d610c5ac78a4742922e2c76c0de7bf"
server_key_version: "encryptionkey@0"
I20260618 22:18:47.686859 10850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/minicluster-data/ts-1/wal/instance:
uuid: "a49d50a39627487cbbc80e1f42fab1ef"
format_stamp: "Formatted at 2026-06-18 22:18:47 on dist-test-slave-g6r4"
server_key: "a700e19d628529d7ed72c71499701721"
server_key_iv: "d1d610c5ac78a4742922e2c76c0de7bf"
server_key_version: "encryptionkey@0"
I20260618 22:18:47.690289 10850 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:18:47.692467 10865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:47.693598 10850 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:18:47.693718 10850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/minicluster-data/ts-1/wal
uuid: "a49d50a39627487cbbc80e1f42fab1ef"
format_stamp: "Formatted at 2026-06-18 22:18:47 on dist-test-slave-g6r4"
server_key: "a700e19d628529d7ed72c71499701721"
server_key_iv: "d1d610c5ac78a4742922e2c76c0de7bf"
server_key_version: "encryptionkey@0"
I20260618 22:18:47.693822 10850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:18:47.723613 10850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:18:47.726915 10850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:18:47.727116 10850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:18:47.727689 10850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:18:47.728493 10850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:18:47.728567 10850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:47.728636 10850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:18:47.728688 10850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:47.737102 10850 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:45419
I20260618 22:18:47.737136 10978 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:45419 every 8 connection(s)
I20260618 22:18:47.737993 10850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
I20260618 22:18:47.745654 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 10850
I20260618 22:18:47.745788 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/minicluster-data/ts-1/wal/instance
I20260618 22:18:47.746119 393 external_mini_cluster.cc:1428] Setting key 8d2acbb748af03fdc758ed3eb35a3d0b
Jun 18 22:18:47 dist-test-slave-g6r4 krb5kdc[10609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821127, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:18:47.751121 10693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:47.739635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:46509 (local address 127.0.98.126:41859)
0618 22:18:47.739766 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:47.739771 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:47.740620 (+ 849us) server_negotiation.cc:407] Connection header received
0618 22:18:47.741810 (+ 1190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:47.741812 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:47.741856 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:47.741990 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:47.743066 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:47.743580 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:47.744622 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:47.744861 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:47.744935 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:47.747469 (+ 2534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:47.747472 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:47.747475 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:47.747502 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:47.749027 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:47.749624 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:47.749627 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:47.749628 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:47.749703 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:47.750116 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:47.750118 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:47.750119 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:47.750279 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:47.750360 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:47.750928 (+ 568us) server_negotiation.cc:299] Negotiation successful
0618 22:18:47.751015 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260618 22:18:47.752305 10981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:47.740058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:41859 (local address 127.0.98.66:46509)
0618 22:18:47.740477 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:47.740519 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:47.741622 (+ 1103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:47.742132 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:47.742139 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:47.742476 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:47.742943 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:47.742957 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:47.743682 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:47.743685 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:47.744522 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:47.744530 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:47.744980 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:47.744983 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:47.745110 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:47.745660 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:47.745675 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:47.747328 (+ 1653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:18:47.749139 (+ 1811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:47.749145 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:47.749160 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:47.749517 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:47.749795 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:47.749800 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:47.749804 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:47.750024 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:47.750390 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:18:47.750399 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:18:47.750694 (+ 295us) client_negotiation.cc:769] Sending connection context
0618 22:18:47.750895 (+ 201us) client_negotiation.cc:240] Negotiation successful
0618 22:18:47.751238 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":97,"threads_started":1}
I20260618 22:18:47.753511 10979 heartbeater.cc:347] Connected to a master server at 127.0.98.126:41859
I20260618 22:18:47.753814 10979 heartbeater.cc:464] Registering TS with master...
I20260618 22:18:47.754411 10979 heartbeater.cc:511] Master 127.0.98.126:41859 requested a full tablet report, sending...
I20260618 22:18:47.755551 10654 ts_manager.cc:194] Registered new tserver with Master: a49d50a39627487cbbc80e1f42fab1ef (127.0.98.66:45419)
WARNING: no policy specified for kudu/127.0.98.67@KRBTEST.COM; defaulting to no policy
I20260618 22:18:47.756194 10654 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.66@KRBTEST.COM'} at 127.0.98.66:46509
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab.
I20260618 22:18:47.772706 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:41859
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:39035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:18:47.879906 10984 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:18:47.880165 10984 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:18:47.880242 10984 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:18:47.884325 10984 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:18:47.884414 10984 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:18:47.884500 10984 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260618 22:18:47.889524 10984 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:39035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:41859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10984
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:18:47.890565 10984 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:18:47.891530 10984 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:18:47.900653 10992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:47.900709 10984 server_base.cc:1034] running on GCE node
W20260618 22:18:47.900664 10990 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:18:47.900664 10989 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:18:47.901280 10984 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:18:47.901934 10984 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:18:47.903106 10984 hybrid_clock.cc:648] HybridClock initialized: now 1781821127903094 us; error 26 us; skew 500 ppm
Jun 18 22:18:47 dist-test-slave-g6r4 krb5kdc[10609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821127, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:18:47.905628 10984 init.cc:377] Logged in from keytab as kudu/127.0.98.67@KRBTEST.COM (short username kudu)
I20260618 22:18:47.906458 10984 webserver.cc:460] Webserver started at http://127.0.98.67:40715/ using document root <none> and password file <none>
I20260618 22:18:47.906992 10984 fs_manager.cc:359] Metadata directory not provided
I20260618 22:18:47.907079 10984 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:18:47.907275 10984 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:18:47.909168 10984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/minicluster-data/ts-2/data/instance:
uuid: "d76da58c91ba40998f3556e3d22a889d"
format_stamp: "Formatted at 2026-06-18 22:18:47 on dist-test-slave-g6r4"
server_key: "a4a862dd149e0074f5f1700ddb4c4222"
server_key_iv: "9c6346ec402f9e5971c0952af3169faf"
server_key_version: "encryptionkey@0"
I20260618 22:18:47.909706 10984 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/minicluster-data/ts-2/wal/instance:
uuid: "d76da58c91ba40998f3556e3d22a889d"
format_stamp: "Formatted at 2026-06-18 22:18:47 on dist-test-slave-g6r4"
server_key: "a4a862dd149e0074f5f1700ddb4c4222"
server_key_iv: "9c6346ec402f9e5971c0952af3169faf"
server_key_version: "encryptionkey@0"
I20260618 22:18:47.913131 10984 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260618 22:18:47.915376 10999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:47.916568 10984 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260618 22:18:47.916653 10984 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/minicluster-data/ts-2/wal
uuid: "d76da58c91ba40998f3556e3d22a889d"
format_stamp: "Formatted at 2026-06-18 22:18:47 on dist-test-slave-g6r4"
server_key: "a4a862dd149e0074f5f1700ddb4c4222"
server_key_iv: "9c6346ec402f9e5971c0952af3169faf"
server_key_version: "encryptionkey@0"
I20260618 22:18:47.916774 10984 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:18:47.929338 10984 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:18:47.932263 10984 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:18:47.932403 10984 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:18:47.932901 10984 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:18:47.933612 10984 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:18:47.933657 10984 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:47.933696 10984 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:18:47.933710 10984 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:18:47.942047 10984 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:38375
I20260618 22:18:47.942111 11112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:38375 every 8 connection(s)
I20260618 22:18:47.942934 10984 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
I20260618 22:18:47.948277 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 10984
I20260618 22:18:47.948387 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/minicluster-data/ts-2/wal/instance
I20260618 22:18:47.948691 393 external_mini_cluster.cc:1428] Setting key 8e8248f73eb42a5edfdb5a27f1666808
Jun 18 22:18:47 dist-test-slave-g6r4 krb5kdc[10609](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821127, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:18:47.956277 10693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:47.944647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:45877 (local address 127.0.98.126:41859)
0618 22:18:47.944822 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:47.944827 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:47.945580 (+ 753us) server_negotiation.cc:407] Connection header received
0618 22:18:47.946840 (+ 1260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:47.946843 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:47.946897 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:47.946990 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:47.948099 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:47.948895 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:47.950196 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:47.950421 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:47.950533 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:47.953196 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:47.953199 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:47.953201 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:47.953224 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:47.954595 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:47.955049 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:47.955051 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:47.955052 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:47.955120 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:47.955402 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:47.955404 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:47.955405 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:47.955568 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:47.955641 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:47.956069 (+ 428us) server_negotiation.cc:299] Negotiation successful
0618 22:18:47.956163 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260618 22:18:47.957134 11115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:47.945024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:41859 (local address 127.0.98.67:45877)
0618 22:18:47.945432 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:47.945471 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:47.946572 (+ 1101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:47.947135 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:47.947141 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:47.947477 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:18:47.947946 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:47.947960 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:47.949047 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:47.949049 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:47.950076 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:47.950085 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:47.950563 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:47.950567 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:47.950727 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:47.951275 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:18:47.951291 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:18:47.953051 (+ 1760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:18:47.954696 (+ 1645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:47.954701 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:47.954711 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:47.954962 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:47.955197 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:18:47.955199 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:18:47.955200 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:18:47.955323 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:18:47.955653 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:18:47.955659 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:18:47.955869 (+ 210us) client_negotiation.cc:769] Sending connection context
0618 22:18:47.956024 (+ 155us) client_negotiation.cc:240] Negotiation successful
0618 22:18:47.956256 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":108,"threads_started":1}
I20260618 22:18:47.958334 11113 heartbeater.cc:347] Connected to a master server at 127.0.98.126:41859
I20260618 22:18:47.958573 11113 heartbeater.cc:464] Registering TS with master...
I20260618 22:18:47.959025 11113 heartbeater.cc:511] Master 127.0.98.126:41859 requested a full tablet report, sending...
I20260618 22:18:47.960057 10654 ts_manager.cc:194] Registered new tserver with Master: d76da58c91ba40998f3556e3d22a889d (127.0.98.67:38375)
I20260618 22:18:47.960580 10654 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.67@KRBTEST.COM'} at 127.0.98.67:45877
I20260618 22:18:47.962594 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 22:18:47.971437 10693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:47.964402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53978 (local address 127.0.98.126:41859)
0618 22:18:47.964509 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:47.964512 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:47.964711 (+ 199us) server_negotiation.cc:407] Connection header received
0618 22:18:47.964793 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:47.964795 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:47.964828 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:47.964905 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:18:47.965532 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:47.966067 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:47.966873 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:47.967086 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:47.967165 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:47.968450 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:18:47.968453 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:18:47.968455 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:18:47.968478 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:18:47.970034 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:47.970419 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:47.970422 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:47.970423 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:47.970499 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:18:47.970719 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:18:47.970721 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:18:47.970722 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:18:47.970956 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:18:47.971062 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:47.971236 (+ 174us) server_negotiation.cc:299] Negotiation successful
0618 22:18:47.971327 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260618 22:18:47.974129 10654 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53978:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 22:18:47.977180 10654 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:18:47.991391 11125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:47.985485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:38375 (local address 127.0.0.1:60278)
0618 22:18:47.985990 (+ 505us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:47.986004 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:47.986096 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:47.987271 (+ 1175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:47.987274 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:47.987290 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:18:47.987376 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:47.987384 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:47.988420 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:47.988426 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:47.990035 (+ 1609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:47.990045 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:47.990872 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:47.990875 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:47.990979 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:47.991014 (+ 35us) client_negotiation.cc:769] Sending connection context
0618 22:18:47.991120 (+ 106us) client_negotiation.cc:240] Negotiation successful
0618 22:18:47.991235 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"mutex_wait_us":124,"thread_start_us":103,"threads_started":1}
I20260618 22:18:47.991430 11129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:47.986535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:45419 (local address 127.0.0.1:44350)
0618 22:18:47.986807 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:47.986822 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:47.986912 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:47.987528 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:47.987533 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:47.987549 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:18:47.987631 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:47.987637 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:47.988957 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:47.988964 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:47.990115 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:47.990128 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:47.991128 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:47.991131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:47.991205 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:47.991224 (+ 19us) client_negotiation.cc:769] Sending connection context
0618 22:18:47.991280 (+ 56us) client_negotiation.cc:240] Negotiation successful
0618 22:18:47.991332 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":1,"thread_start_us":92,"threads_started":1}
I20260618 22:18:47.991719 11124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:47.985301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:43243 (local address 127.0.0.1:55484)
0618 22:18:47.985646 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:47.985697 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:47.985860 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:47.986868 (+ 1008us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:47.986875 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:47.986896 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:18:47.987041 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:47.987049 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:47.988616 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:47.988619 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:47.990035 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:47.990045 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:47.991326 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:47.991330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:47.991413 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:47.991434 (+ 21us) client_negotiation.cc:769] Sending connection context
0618 22:18:47.991504 (+ 70us) client_negotiation.cc:240] Negotiation successful
0618 22:18:47.991577 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":143,"thread_start_us":68,"threads_started":1}
I20260618 22:18:47.991801 11127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:47.986189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44350 (local address 127.0.98.66:45419)
0618 22:18:47.986524 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:47.986534 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:47.986998 (+ 464us) server_negotiation.cc:407] Connection header received
0618 22:18:47.987066 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:47.987071 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:47.987236 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:47.987358 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:18:47.987759 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:47.988835 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:47.990240 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:47.991023 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:47.991130 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:47.991262 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:47.991316 (+ 54us) server_negotiation.cc:299] Negotiation successful
0618 22:18:47.991406 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":122,"threads_started":1}
I20260618 22:18:47.991995 11128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:47.986190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60278 (local address 127.0.98.67:38375)
0618 22:18:47.986809 (+ 619us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:47.986827 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:47.986853 (+ 26us) server_negotiation.cc:407] Connection header received
0618 22:18:47.986927 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:47.986934 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:47.987129 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:47.987281 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:18:47.987478 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:47.988290 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:47.990161 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:47.990743 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:47.990876 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:47.991032 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:47.991734 (+ 702us) server_negotiation.cc:299] Negotiation successful
0618 22:18:47.991851 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":167,"threads_started":1}
I20260618 22:18:47.992228 11126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:47.985692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55484 (local address 127.0.98.65:43243)
0618 22:18:47.986300 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:47.986318 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:47.986360 (+ 42us) server_negotiation.cc:407] Connection header received
0618 22:18:47.986451 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:47.986457 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:47.986687 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:47.987184 (+ 497us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:18:47.987356 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:47.988477 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:47.990202 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:47.991023 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:47.991130 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:47.991262 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:47.992008 (+ 746us) server_negotiation.cc:299] Negotiation successful
0618 22:18:47.992101 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":69,"threads_started":1}
I20260618 22:18:47.994589 11047 tablet_service.cc:1467] Processing CreateTablet for tablet b34e9c692b9b406da32ba339430452d9 (DEFAULT_TABLE table=test-table [id=e1b39e012dc849c09c22e9e79c6055c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:18:47.994640 10779 tablet_service.cc:1467] Processing CreateTablet for tablet b34e9c692b9b406da32ba339430452d9 (DEFAULT_TABLE table=test-table [id=e1b39e012dc849c09c22e9e79c6055c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:18:47.994637 10913 tablet_service.cc:1467] Processing CreateTablet for tablet b34e9c692b9b406da32ba339430452d9 (DEFAULT_TABLE table=test-table [id=e1b39e012dc849c09c22e9e79c6055c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:18:47.995555 11047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b34e9c692b9b406da32ba339430452d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:18:47.995600 10779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b34e9c692b9b406da32ba339430452d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:18:47.995600 10913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b34e9c692b9b406da32ba339430452d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:18:48.001863 11130 tablet_bootstrap.cc:492] T b34e9c692b9b406da32ba339430452d9 P d76da58c91ba40998f3556e3d22a889d: Bootstrap starting.
I20260618 22:18:48.003095 11132 tablet_bootstrap.cc:492] T b34e9c692b9b406da32ba339430452d9 P a49d50a39627487cbbc80e1f42fab1ef: Bootstrap starting.
I20260618 22:18:48.004338 11130 tablet_bootstrap.cc:654] T b34e9c692b9b406da32ba339430452d9 P d76da58c91ba40998f3556e3d22a889d: Neither blocks nor log segments found. Creating new log.
I20260618 22:18:48.005065 11132 tablet_bootstrap.cc:654] T b34e9c692b9b406da32ba339430452d9 P a49d50a39627487cbbc80e1f42fab1ef: Neither blocks nor log segments found. Creating new log.
I20260618 22:18:48.005050 11130 log.cc:824] T b34e9c692b9b406da32ba339430452d9 P d76da58c91ba40998f3556e3d22a889d: Log is configured to *not* fsync() on all Append() calls
I20260618 22:18:48.005661 11131 tablet_bootstrap.cc:492] T b34e9c692b9b406da32ba339430452d9 P 3f37594f859841eab5768680ec166be4: Bootstrap starting.
I20260618 22:18:48.005870 11132 log.cc:824] T b34e9c692b9b406da32ba339430452d9 P a49d50a39627487cbbc80e1f42fab1ef: Log is configured to *not* fsync() on all Append() calls
I20260618 22:18:48.007167 11130 tablet_bootstrap.cc:492] T b34e9c692b9b406da32ba339430452d9 P d76da58c91ba40998f3556e3d22a889d: No bootstrap required, opened a new log
I20260618 22:18:48.007356 11130 ts_tablet_manager.cc:1397] T b34e9c692b9b406da32ba339430452d9 P d76da58c91ba40998f3556e3d22a889d: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260618 22:18:48.007705 11131 tablet_bootstrap.cc:654] T b34e9c692b9b406da32ba339430452d9 P 3f37594f859841eab5768680ec166be4: Neither blocks nor log segments found. Creating new log.
I20260618 22:18:48.007762 11132 tablet_bootstrap.cc:492] T b34e9c692b9b406da32ba339430452d9 P a49d50a39627487cbbc80e1f42fab1ef: No bootstrap required, opened a new log
I20260618 22:18:48.007925 11132 ts_tablet_manager.cc:1397] T b34e9c692b9b406da32ba339430452d9 P a49d50a39627487cbbc80e1f42fab1ef: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260618 22:18:48.008652 11131 log.cc:824] T b34e9c692b9b406da32ba339430452d9 P 3f37594f859841eab5768680ec166be4: Log is configured to *not* fsync() on all Append() calls
I20260618 22:18:48.010258 11131 tablet_bootstrap.cc:492] T b34e9c692b9b406da32ba339430452d9 P 3f37594f859841eab5768680ec166be4: No bootstrap required, opened a new log
I20260618 22:18:48.010444 11131 ts_tablet_manager.cc:1397] T b34e9c692b9b406da32ba339430452d9 P 3f37594f859841eab5768680ec166be4: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260618 22:18:48.011206 11130 raft_consensus.cc:348] T b34e9c692b9b406da32ba339430452d9 P d76da58c91ba40998f3556e3d22a889d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f37594f859841eab5768680ec166be4" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43243 } } peers { permanent_uuid: "d76da58c91ba40998f3556e3d22a889d" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38375 } } peers { permanent_uuid: "a49d50a39627487cbbc80e1f42fab1ef" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 45419 } }
I20260618 22:18:48.011489 11130 raft_consensus.cc:374] T b34e9c692b9b406da32ba339430452d9 P d76da58c91ba40998f3556e3d22a889d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:18:48.011576 11130 raft_consensus.cc:729] T b34e9c692b9b406da32ba339430452d9 P d76da58c91ba40998f3556e3d22a889d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d76da58c91ba40998f3556e3d22a889d, State: Initialized, Role: FOLLOWER
I20260618 22:18:48.011549 11132 raft_consensus.cc:348] T b34e9c692b9b406da32ba339430452d9 P a49d50a39627487cbbc80e1f42fab1ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f37594f859841eab5768680ec166be4" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43243 } } peers { permanent_uuid: "d76da58c91ba40998f3556e3d22a889d" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38375 } } peers { permanent_uuid: "a49d50a39627487cbbc80e1f42fab1ef" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 45419 } }
I20260618 22:18:48.011801 11132 raft_consensus.cc:374] T b34e9c692b9b406da32ba339430452d9 P a49d50a39627487cbbc80e1f42fab1ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:18:48.011863 11132 raft_consensus.cc:729] T b34e9c692b9b406da32ba339430452d9 P a49d50a39627487cbbc80e1f42fab1ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a49d50a39627487cbbc80e1f42fab1ef, State: Initialized, Role: FOLLOWER
I20260618 22:18:48.012094 11130 consensus_queue.cc:260] T b34e9c692b9b406da32ba339430452d9 P d76da58c91ba40998f3556e3d22a889d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f37594f859841eab5768680ec166be4" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43243 } } peers { permanent_uuid: "d76da58c91ba40998f3556e3d22a889d" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38375 } } peers { permanent_uuid: "a49d50a39627487cbbc80e1f42fab1ef" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 45419 } }
I20260618 22:18:48.012275 11132 consensus_queue.cc:260] T b34e9c692b9b406da32ba339430452d9 P a49d50a39627487cbbc80e1f42fab1ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f37594f859841eab5768680ec166be4" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43243 } } peers { permanent_uuid: "d76da58c91ba40998f3556e3d22a889d" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38375 } } peers { permanent_uuid: "a49d50a39627487cbbc80e1f42fab1ef" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 45419 } }
I20260618 22:18:48.012768 11113 heartbeater.cc:503] Master 127.0.98.126:41859 was elected leader, sending a full tablet report...
I20260618 22:18:48.013193 11130 ts_tablet_manager.cc:1428] T b34e9c692b9b406da32ba339430452d9 P d76da58c91ba40998f3556e3d22a889d: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260618 22:18:48.013193 11132 ts_tablet_manager.cc:1428] T b34e9c692b9b406da32ba339430452d9 P a49d50a39627487cbbc80e1f42fab1ef: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260618 22:18:48.013770 11131 raft_consensus.cc:348] T b34e9c692b9b406da32ba339430452d9 P 3f37594f859841eab5768680ec166be4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f37594f859841eab5768680ec166be4" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43243 } } peers { permanent_uuid: "d76da58c91ba40998f3556e3d22a889d" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38375 } } peers { permanent_uuid: "a49d50a39627487cbbc80e1f42fab1ef" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 45419 } }
I20260618 22:18:48.014078 10979 heartbeater.cc:503] Master 127.0.98.126:41859 was elected leader, sending a full tablet report...
I20260618 22:18:48.014117 11131 raft_consensus.cc:374] T b34e9c692b9b406da32ba339430452d9 P 3f37594f859841eab5768680ec166be4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:18:48.014178 11131 raft_consensus.cc:729] T b34e9c692b9b406da32ba339430452d9 P 3f37594f859841eab5768680ec166be4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f37594f859841eab5768680ec166be4, State: Initialized, Role: FOLLOWER
I20260618 22:18:48.014561 11131 consensus_queue.cc:260] T b34e9c692b9b406da32ba339430452d9 P 3f37594f859841eab5768680ec166be4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f37594f859841eab5768680ec166be4" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43243 } } peers { permanent_uuid: "d76da58c91ba40998f3556e3d22a889d" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38375 } } peers { permanent_uuid: "a49d50a39627487cbbc80e1f42fab1ef" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 45419 } }
I20260618 22:18:48.015123 10845 heartbeater.cc:503] Master 127.0.98.126:41859 was elected leader, sending a full tablet report...
I20260618 22:18:48.015461 11131 ts_tablet_manager.cc:1428] T b34e9c692b9b406da32ba339430452d9 P 3f37594f859841eab5768680ec166be4: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260618 22:18:48.193948 11114 tablet.cc:2369] T b34e9c692b9b406da32ba339430452d9 P d76da58c91ba40998f3556e3d22a889d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260618 22:18:48.239035 10980 tablet.cc:2369] T b34e9c692b9b406da32ba339430452d9 P a49d50a39627487cbbc80e1f42fab1ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260618 22:18:48.263199 10846 tablet.cc:2369] T b34e9c692b9b406da32ba339430452d9 P 3f37594f859841eab5768680ec166be4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:18:48.452808 11136 raft_consensus.cc:482] T b34e9c692b9b406da32ba339430452d9 P d76da58c91ba40998f3556e3d22a889d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:18:48.453089 11136 raft_consensus.cc:504] T b34e9c692b9b406da32ba339430452d9 P d76da58c91ba40998f3556e3d22a889d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f37594f859841eab5768680ec166be4" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43243 } } peers { permanent_uuid: "d76da58c91ba40998f3556e3d22a889d" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38375 } } peers { permanent_uuid: "a49d50a39627487cbbc80e1f42fab1ef" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 45419 } }
I20260618 22:18:48.454293 11136 leader_election.cc:288] T b34e9c692b9b406da32ba339430452d9 P d76da58c91ba40998f3556e3d22a889d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3f37594f859841eab5768680ec166be4 (127.0.98.65:43243), a49d50a39627487cbbc80e1f42fab1ef (127.0.98.66:45419)
I20260618 22:18:48.458352 11115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:48.454512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:43243 (local address 127.0.98.67:49333)
0618 22:18:48.454736 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:48.454753 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:48.454852 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:48.455176 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:48.455178 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:48.455197 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:18:48.455326 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:48.455332 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:48.456238 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:48.456242 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:48.457147 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:48.457156 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:48.457928 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:48.457931 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:48.458037 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:48.458060 (+ 23us) client_negotiation.cc:769] Sending connection context
0618 22:18:48.458126 (+ 66us) client_negotiation.cc:240] Negotiation successful
0618 22:18:48.458212 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":96}
I20260618 22:18:48.458464 11126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:48.454618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:49333 (local address 127.0.98.65:43243)
0618 22:18:48.454833 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:48.454839 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:48.454868 (+ 29us) server_negotiation.cc:407] Connection header received
0618 22:18:48.454973 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:48.454978 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:48.455050 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:48.455133 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:18:48.455437 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:48.456097 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:48.457268 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:48.457768 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:48.457862 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:48.457984 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:48.458202 (+ 218us) server_negotiation.cc:299] Negotiation successful
0618 22:18:48.458283 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260618 22:18:48.458508 11127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:48.454714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:37199 (local address 127.0.98.66:45419)
0618 22:18:48.454828 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:48.454832 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:48.455072 (+ 240us) server_negotiation.cc:407] Connection header received
0618 22:18:48.455197 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:48.455200 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:48.455248 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:48.455330 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:18:48.455627 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:48.456095 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:48.457430 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:48.457799 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:48.457945 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:48.458006 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:48.458287 (+ 281us) server_negotiation.cc:299] Negotiation successful
0618 22:18:48.458373 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260618 22:18:48.458631 11139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:48.454612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:45419 (local address 127.0.98.67:37199)
0618 22:18:48.454965 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:48.454979 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:18:48.455088 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:18:48.455352 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:18:48.455354 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:18:48.455369 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:18:48.455513 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:48.455520 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:48.456207 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:48.456211 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:48.457242 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:18:48.457255 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:18:48.457963 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:18:48.457966 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:18:48.458079 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:48.458097 (+ 18us) client_negotiation.cc:769] Sending connection context
0618 22:18:48.458243 (+ 146us) client_negotiation.cc:240] Negotiation successful
0618 22:18:48.458321 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"mutex_wait_us":16,"thread_start_us":89,"threads_started":1}
I20260618 22:18:48.459270 10799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b34e9c692b9b406da32ba339430452d9" candidate_uuid: "d76da58c91ba40998f3556e3d22a889d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f37594f859841eab5768680ec166be4" is_pre_election: true
I20260618 22:18:48.459288 10933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b34e9c692b9b406da32ba339430452d9" candidate_uuid: "d76da58c91ba40998f3556e3d22a889d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a49d50a39627487cbbc80e1f42fab1ef" is_pre_election: true
I20260618 22:18:48.459594 10799 raft_consensus.cc:2454] T b34e9c692b9b406da32ba339430452d9 P 3f37594f859841eab5768680ec166be4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d76da58c91ba40998f3556e3d22a889d in term 0.
I20260618 22:18:48.459594 10933 raft_consensus.cc:2454] T b34e9c692b9b406da32ba339430452d9 P a49d50a39627487cbbc80e1f42fab1ef [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d76da58c91ba40998f3556e3d22a889d in term 0.
I20260618 22:18:48.460076 11002 leader_election.cc:302] T b34e9c692b9b406da32ba339430452d9 P d76da58c91ba40998f3556e3d22a889d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3f37594f859841eab5768680ec166be4, a49d50a39627487cbbc80e1f42fab1ef, d76da58c91ba40998f3556e3d22a889d; no voters:
I20260618 22:18:48.460399 11136 raft_consensus.cc:2781] T b34e9c692b9b406da32ba339430452d9 P d76da58c91ba40998f3556e3d22a889d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:18:48.460491 11136 raft_consensus.cc:482] T b34e9c692b9b406da32ba339430452d9 P d76da58c91ba40998f3556e3d22a889d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:18:48.460523 11136 raft_consensus.cc:3037] T b34e9c692b9b406da32ba339430452d9 P d76da58c91ba40998f3556e3d22a889d [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:18:48.461537 11136 raft_consensus.cc:504] T b34e9c692b9b406da32ba339430452d9 P d76da58c91ba40998f3556e3d22a889d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f37594f859841eab5768680ec166be4" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43243 } } peers { permanent_uuid: "d76da58c91ba40998f3556e3d22a889d" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38375 } } peers { permanent_uuid: "a49d50a39627487cbbc80e1f42fab1ef" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 45419 } }
I20260618 22:18:48.461874 11136 leader_election.cc:288] T b34e9c692b9b406da32ba339430452d9 P d76da58c91ba40998f3556e3d22a889d [CANDIDATE]: Term 1 election: Requested vote from peers 3f37594f859841eab5768680ec166be4 (127.0.98.65:43243), a49d50a39627487cbbc80e1f42fab1ef (127.0.98.66:45419)
I20260618 22:18:48.462251 10933 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b34e9c692b9b406da32ba339430452d9" candidate_uuid: "d76da58c91ba40998f3556e3d22a889d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a49d50a39627487cbbc80e1f42fab1ef"
I20260618 22:18:48.462271 10799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b34e9c692b9b406da32ba339430452d9" candidate_uuid: "d76da58c91ba40998f3556e3d22a889d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f37594f859841eab5768680ec166be4"
I20260618 22:18:48.462378 10933 raft_consensus.cc:3037] T b34e9c692b9b406da32ba339430452d9 P a49d50a39627487cbbc80e1f42fab1ef [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:18:48.462450 10799 raft_consensus.cc:3037] T b34e9c692b9b406da32ba339430452d9 P 3f37594f859841eab5768680ec166be4 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:18:48.463436 10933 raft_consensus.cc:2454] T b34e9c692b9b406da32ba339430452d9 P a49d50a39627487cbbc80e1f42fab1ef [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d76da58c91ba40998f3556e3d22a889d in term 1.
I20260618 22:18:48.463657 10799 raft_consensus.cc:2454] T b34e9c692b9b406da32ba339430452d9 P 3f37594f859841eab5768680ec166be4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d76da58c91ba40998f3556e3d22a889d in term 1.
I20260618 22:18:48.463766 11003 leader_election.cc:302] T b34e9c692b9b406da32ba339430452d9 P d76da58c91ba40998f3556e3d22a889d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a49d50a39627487cbbc80e1f42fab1ef, d76da58c91ba40998f3556e3d22a889d; no voters:
I20260618 22:18:48.464011 11136 raft_consensus.cc:2781] T b34e9c692b9b406da32ba339430452d9 P d76da58c91ba40998f3556e3d22a889d [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:18:48.464236 11136 raft_consensus.cc:686] T b34e9c692b9b406da32ba339430452d9 P d76da58c91ba40998f3556e3d22a889d [term 1 LEADER]: Becoming Leader. State: Replica: d76da58c91ba40998f3556e3d22a889d, State: Running, Role: LEADER
I20260618 22:18:48.464560 11136 consensus_queue.cc:237] T b34e9c692b9b406da32ba339430452d9 P d76da58c91ba40998f3556e3d22a889d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f37594f859841eab5768680ec166be4" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43243 } } peers { permanent_uuid: "d76da58c91ba40998f3556e3d22a889d" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38375 } } peers { permanent_uuid: "a49d50a39627487cbbc80e1f42fab1ef" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 45419 } }
I20260618 22:18:48.467808 10654 catalog_manager.cc:5495] T b34e9c692b9b406da32ba339430452d9 P d76da58c91ba40998f3556e3d22a889d reported cstate change: term changed from 0 to 1, leader changed from <none> to d76da58c91ba40998f3556e3d22a889d (127.0.98.67). New cstate: current_term: 1 leader_uuid: "d76da58c91ba40998f3556e3d22a889d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f37594f859841eab5768680ec166be4" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d76da58c91ba40998f3556e3d22a889d" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 38375 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a49d50a39627487cbbc80e1f42fab1ef" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 45419 } health_report { overall_health: UNKNOWN } } }
I20260618 22:18:48.552603 11144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:48.549057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60284 (local address 127.0.98.67:38375)
0618 22:18:48.549271 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0618 22:18:48.549274 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:18:48.549288 (+ 14us) server_negotiation.cc:407] Connection header received
0618 22:18:48.549460 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:18:48.549463 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:18:48.549516 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:18:48.549614 (+ 98us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:18:48.549842 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:48.550466 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:48.551303 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:18:48.551523 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:18:48.551599 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:18:48.551854 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:18:48.552265 (+ 411us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:18:48.552356 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0618 22:18:48.552426 (+ 70us) server_negotiation.cc:299] Negotiation successful
0618 22:18:48.552470 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":55,"threads_started":1}
I20260618 22:18:48.558197 10933 raft_consensus.cc:1261] T b34e9c692b9b406da32ba339430452d9 P a49d50a39627487cbbc80e1f42fab1ef [term 1 FOLLOWER]: Refusing update from remote peer d76da58c91ba40998f3556e3d22a889d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:18:48.558197 10799 raft_consensus.cc:1261] T b34e9c692b9b406da32ba339430452d9 P 3f37594f859841eab5768680ec166be4 [term 1 FOLLOWER]: Refusing update from remote peer d76da58c91ba40998f3556e3d22a889d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:18:48.558941 11140 consensus_queue.cc:1035] T b34e9c692b9b406da32ba339430452d9 P d76da58c91ba40998f3556e3d22a889d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f37594f859841eab5768680ec166be4" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 43243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:18:48.559151 11136 consensus_queue.cc:1035] T b34e9c692b9b406da32ba339430452d9 P d76da58c91ba40998f3556e3d22a889d [LEADER]: Connected to new peer: Peer: permanent_uuid: "a49d50a39627487cbbc80e1f42fab1ef" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 45419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:18:48.574020 10654 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53978:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:18:48.574162 10654 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53978:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:18:48.576529 10654 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 641152b53bf24f16b1e900c782a05a38: Sending DeleteTablet for 3 replicas of tablet b34e9c692b9b406da32ba339430452d9
I20260618 22:18:48.577201 10779 tablet_service.cc:1514] Processing DeleteTablet for tablet b34e9c692b9b406da32ba339430452d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:18:48 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:55484
I20260618 22:18:48.577240 10913 tablet_service.cc:1514] Processing DeleteTablet for tablet b34e9c692b9b406da32ba339430452d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:18:48 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:44350
I20260618 22:18:48.577842 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 10716
I20260618 22:18:48.577819 11047 tablet_service.cc:1514] Processing DeleteTablet for tablet b34e9c692b9b406da32ba339430452d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:18:48 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:60278
I20260618 22:18:48.578528 11156 tablet_replica.cc:330] T b34e9c692b9b406da32ba339430452d9 P a49d50a39627487cbbc80e1f42fab1ef: stopping tablet replica
I20260618 22:18:48.578823 11156 raft_consensus.cc:2229] T b34e9c692b9b406da32ba339430452d9 P a49d50a39627487cbbc80e1f42fab1ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:18:48.579150 11156 raft_consensus.cc:2258] T b34e9c692b9b406da32ba339430452d9 P a49d50a39627487cbbc80e1f42fab1ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:18:48.579507 11158 tablet_replica.cc:330] T b34e9c692b9b406da32ba339430452d9 P d76da58c91ba40998f3556e3d22a889d: stopping tablet replica
I20260618 22:18:48.579749 11158 raft_consensus.cc:2229] T b34e9c692b9b406da32ba339430452d9 P d76da58c91ba40998f3556e3d22a889d [term 1 LEADER]: Raft consensus shutting down.
I20260618 22:18:48.580047 11158 raft_consensus.cc:2258] T b34e9c692b9b406da32ba339430452d9 P d76da58c91ba40998f3556e3d22a889d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:18:48.580427 11156 ts_tablet_manager.cc:1905] T b34e9c692b9b406da32ba339430452d9 P a49d50a39627487cbbc80e1f42fab1ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:18:48.581110 11158 ts_tablet_manager.cc:1905] T b34e9c692b9b406da32ba339430452d9 P d76da58c91ba40998f3556e3d22a889d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:18:48.583050 11156 ts_tablet_manager.cc:1918] T b34e9c692b9b406da32ba339430452d9 P a49d50a39627487cbbc80e1f42fab1ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:18:48.583184 11156 log.cc:1196] T b34e9c692b9b406da32ba339430452d9 P a49d50a39627487cbbc80e1f42fab1ef: Deleting WAL directory at /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/minicluster-data/ts-1/wal/wals/b34e9c692b9b406da32ba339430452d9
I20260618 22:18:48.583539 11156 ts_tablet_manager.cc:1939] T b34e9c692b9b406da32ba339430452d9 P a49d50a39627487cbbc80e1f42fab1ef: Deleting consensus metadata
I20260618 22:18:48.583659 11158 ts_tablet_manager.cc:1918] T b34e9c692b9b406da32ba339430452d9 P d76da58c91ba40998f3556e3d22a889d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:18:48.583808 11158 log.cc:1196] T b34e9c692b9b406da32ba339430452d9 P d76da58c91ba40998f3556e3d22a889d: Deleting WAL directory at /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781821065380026-393-0/minicluster-data/ts-2/wal/wals/b34e9c692b9b406da32ba339430452d9
I20260618 22:18:48.584142 11158 ts_tablet_manager.cc:1939] T b34e9c692b9b406da32ba339430452d9 P d76da58c91ba40998f3556e3d22a889d: Deleting consensus metadata
I20260618 22:18:48.584774 10641 catalog_manager.cc:4841] TS a49d50a39627487cbbc80e1f42fab1ef (127.0.98.66:45419): tablet b34e9c692b9b406da32ba339430452d9 (table test-table [id=e1b39e012dc849c09c22e9e79c6055c8]) successfully deleted
W20260618 22:18:48.584950 10640 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.65:43243 (error 108)
I20260618 22:18:48.585201 10641 catalog_manager.cc:4841] TS d76da58c91ba40998f3556e3d22a889d (127.0.98.67:38375): tablet b34e9c692b9b406da32ba339430452d9 (table test-table [id=e1b39e012dc849c09c22e9e79c6055c8]) successfully deleted
I20260618 22:18:48.585985 11160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:18:48.585562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:43243 (local address 127.0.0.1:55492)
0618 22:18:48.585765 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0618 22:18:48.585837 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.65:43243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":69,"threads_started":1}
I20260618 22:18:48.586057 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 10850
W20260618 22:18:48.586148 10640 catalog_manager.cc:4584] TS 3f37594f859841eab5768680ec166be4 (127.0.98.65:43243): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b34e9c692b9b406da32ba339430452d9: Network error: Client connection negotiation failed: client connection to 127.0.98.65:43243: connect: Connection refused (error 111)
I20260618 22:18:48.592420 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 10984
I20260618 22:18:48.597635 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 10622
2026-06-18T22:18:48Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2695 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:18:48 dist-test-slave-g6r4 krb5kdc[11163](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:18:48 dist-test-slave-g6r4 krb5kdc[11163](info): set up 2 sockets
Jun 18 22:18:48 dist-test-slave-g6r4 krb5kdc[11163](info): commencing operation
krb5kdc: starting...
W20260618 22:18:49.650942 393 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/krb5kdc/test-user.keytab.
Jun 18 22:18:49 dist-test-slave-g6r4 krb5kdc[11163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821129, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:18:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:18:49Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/krb5kdc/rangeradmin_127.0.98.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/krb5kdc/rangeradmin_127.0.98.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/krb5kdc/rangerlookup_127.0.98.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/krb5kdc/rangerlookup_127.0.98.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/krb5kdc/HTTP_127.0.98.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/krb5kdc/HTTP_127.0.98.84@KRBTEST.COM.keytab.
I20260618 22:18:49.810894 393 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskIMAyAx/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/postgres -l logfile start
2026-06-18 22:18:52.206 UTC [11193] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-18 22:18:52.206 UTC [11193] LOG: listening on IPv4 address "127.0.98.84", port 54153
2026-06-18 22:18:52.210 UTC [11193] LOG: listening on Unix socket "/tmp/.s.PGSQL.54153"
2026-06-18 22:18:52.244 UTC [11196] LOG: database system was shut down at 2026-06-18 22:18:50 UTC
2026-06-18 22:18:52.248 UTC [11193] LOG: database system is ready to accept connections
I20260618 22:18:53.198565 393 mini_postgres.cc:96] Postgres bound to 54153
2026-06-18 22:18:53.204 UTC [11204] FATAL: database "slave" does not exist
127.0.98.84:54153 - accepting connections
I20260618 22:18:53.205323 393 mini_ranger.cc:161] Starting Ranger...
I20260618 22:18:53.219570 393 mini_ranger.cc:84] Created miniranger Postgres user
I20260618 22:18:53.684523 393 mini_ranger.cc:87] Created ranger Postgres database
I20260618 22:18:53.684602 393 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/ranger-admin
2026-06-18 22:18:53,948 [I] DB FLAVOR :POSTGRES
2026-06-18 22:18:53,949 [I] --------- Verifying Ranger DB connection ---------
2026-06-18 22:18:53,949 [I] Checking connection..
2026-06-18 22:18:53,949 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIMAyAx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIMAyAx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:54153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-18 22:18:54,190 [I] Checking connection passed.
2026-06-18 22:18:54,191 [I] --------- Verifying version history table ---------
2026-06-18 22:18:54,191 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIMAyAx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIMAyAx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:54153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-18 22:18:54,447 [I] Table x_db_version_h does not exist in database ranger
2026-06-18 22:18:54,447 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIMAyAx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIMAyAx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:54153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-18 22:18:54,702 [I] Table x_db_version_h does not exist in database ranger
2026-06-18 22:18:54,703 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-18 22:18:54,703 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIMAyAx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIMAyAx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:54153/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskIMAyAx/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-18 22:18:54.932 UTC [11268] WARNING: there is no transaction in progress
2026-06-18 22:18:54,938 [I] create_dbversion_catalog.sql file imported successfully
2026-06-18 22:18:54,938 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIMAyAx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIMAyAx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:54153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-18 22:18:55,187 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-18 22:18:55,188 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-18 22:18:55,188 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIMAyAx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIMAyAx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:54153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-18 22:18:55,441 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIMAyAx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIMAyAx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:54153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-18 22:18:55,676 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIMAyAx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIMAyAx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:54153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-g6r4.c.gcp-upstream.internal','N') ;"
2026-06-18 22:18:55,910 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIMAyAx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIMAyAx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:54153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-18 22:18:56,148 [I] Table x_portal_user does not exist in database ranger
2026-06-18 22:18:56,148 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIMAyAx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIMAyAx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:54153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-18 22:18:56,377 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-18 22:18:56,377 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIMAyAx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIMAyAx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:54153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-18 22:18:56,629 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIMAyAx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIMAyAx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:54153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-18 22:18:56,862 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-18 22:18:56,862 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIMAyAx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIMAyAx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:54153/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskIMAyAx/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-18 22:18:57.801 UTC [11376] WARNING: there is no transaction in progress
2026-06-18 22:18:57.817 UTC [11376] WARNING: there is no transaction in progress
2026-06-18 22:18:57.833 UTC [11376] WARNING: there is no transaction in progress
2026-06-18 22:18:57.847 UTC [11376] WARNING: there is no transaction in progress
2026-06-18 22:18:57.863 UTC [11376] WARNING: there is no transaction in progress
2026-06-18 22:18:57.882 UTC [11376] WARNING: there is no transaction in progress
2026-06-18 22:18:57.936 UTC [11376] WARNING: there is no transaction in progress
2026-06-18 22:18:57.942 UTC [11376] WARNING: there is no transaction in progress
2026-06-18 22:18:57.957 UTC [11376] WARNING: there is no transaction in progress
2026-06-18 22:18:57.966 UTC [11376] WARNING: there is no transaction in progress
2026-06-18 22:18:57.976 UTC [11376] WARNING: there is no transaction in progress
2026-06-18 22:18:57.990 UTC [11376] WARNING: there is no transaction in progress
2026-06-18 22:18:57.999 UTC [11376] WARNING: there is no transaction in progress
2026-06-18 22:18:58.005 UTC [11376] WARNING: there is no transaction in progress
2026-06-18 22:18:58.011 UTC [11376] WARNING: there is no transaction in progress
2026-06-18 22:18:58.018 UTC [11376] WARNING: there is no transaction in progress
2026-06-18 22:18:58,612 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-18 22:18:58,613 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIMAyAx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIMAyAx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:54153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-18 22:18:58,863 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-18 22:18:58,863 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIMAyAx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIMAyAx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:54153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-18 22:18:59,114 [I] Table x_portal_user already exists in database 'ranger'
2026-06-18 22:18:59,114 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIMAyAx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIMAyAx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:54153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-18 22:18:59,341 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-18 22:18:59,342 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIMAyAx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIMAyAx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:54153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-18 22:18:59,578 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIMAyAx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIMAyAx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:54153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-18 22:18:59,828 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIMAyAx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIMAyAx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:54153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-g6r4.c.gcp-upstream.internal';"
2026-06-18 22:19:00,051 [I] CORE_DB_SCHEMA import status has been updated
2026-06-18 22:19:00,051 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIMAyAx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIMAyAx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:54153/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-18 22:19:00,295 [I] DB_PATCHES have already been applied
I20260618 22:19:00.300981 393 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/ranger-admin:/tmp/dist-test-taskIMAyAx/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskIMAyAx/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskIMAyAx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIMAyAx/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260618 22:19:00.301043 393 mini_ranger.cc:193] Using host: 127.0.98.84
I20260618 22:19:00.304355 393 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.84:34643
Jun 18, 2026 10:19:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 18, 2026 10:19:00 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 18, 2026 10:19:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/ranger-admin/ews/webapp
Jun 18, 2026 10:19:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 18, 2026 10:19:00 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 18, 2026 10:19:01 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/ranger-admin/ews/webapp].
Jun 18, 2026 10:19:01 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260618 22:19:01.418623 393 mini_ranger.cc:160] Time spent starting Ranger: real 8.213s user 0.000s sys 0.010s
Jun 18, 2026 10:19:20 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 18, 2026 10:19:20 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 18, 2026 10:19:20 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 18, 2026 10:19:20 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 18, 2026 10:19:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 18, 2026 10:19:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 18, 2026 10:19:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 18, 2026 10:19:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 18, 2026 10:19:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 18, 2026 10:19:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 18, 2026 10:19:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 18, 2026 10:19:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 18, 2026 10:19:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 18, 2026 10:19:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 18, 2026 10:19:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 18, 2026 10:19:20 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 18, 2026 10:19:20 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 18, 2026 10:19:21 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260618 22:19:22.248237 393 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/krb5kdc/rangerkms_127.0.98.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/krb5kdc/rangerkms_127.0.98.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/krb5kdc/HTTP_127.0.98.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/krb5kdc/HTTP_127.0.98.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 18 22:19:22 dist-test-slave-g6r4 krb5kdc[11163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821162, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260618 22:19:22.325433 393 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260618 22:19:22.334483 393 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260618 22:19:23.143447 393 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260618 22:19:23.143539 393 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/ranger-kms
I20260618 22:19:23.143640 393 mini_ranger_kms.cc:227] Using postgres at 127.0.98.84:54153
2026-06-18 22:19:23,323 [I] DB FLAVOR :POSTGRES
2026-06-18 22:19:23,323 [I] --------- Verifying Ranger DB connection ---------
2026-06-18 22:19:23,323 [I] Checking connection
2026-06-18 22:19:23,323 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIMAyAx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:54153/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-18 22:19:23,566 [I] connection success
2026-06-18 22:19:23,566 [I] --------- Verifying Ranger DB tables ---------
2026-06-18 22:19:23,566 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-18 22:19:23,566 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIMAyAx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:54153/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-18 22:19:23,788 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-18 22:19:23,789 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-18 22:19:23,789 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-18 22:19:23,789 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIMAyAx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:54153/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-18 22:19:24,031 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260618 22:19:24.479166 393 mini_ranger_kms.cc:310] Created kms service
I20260618 22:19:24.681421 393 mini_ranger_kms.cc:326] Created kudu user
I20260618 22:19:24.721407 393 mini_ranger_kms.cc:343] Created rangerkms user
Jun 18 22:19:24 dist-test-slave-g6r4 krb5kdc[11163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821162, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.84@KRBTEST.COM
I20260618 22:19:25.087831 393 mini_ranger_kms.cc:384] Added ranger policy
I20260618 22:19:25.087971 393 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/ranger-kms:/tmp/dist-test-taskIMAyAx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIMAyAx/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskIMAyAx/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260618 22:19:25.087994 393 mini_ranger_kms.cc:240] Using host: 127.0.98.84
I20260618 22:19:25.091900 393 mini_ranger_kms.cc:276] Ranger KMS PID: 11556
I20260618 22:19:25.092015 393 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.84:40803
Jun 18, 2026 10:19:26 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 18, 2026 10:19:26 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 18, 2026 10:19:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 18, 2026 10:19:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 18, 2026 10:19:26 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/ranger-kms/ews/webapp].
Jun 18, 2026 10:19:26 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260618 22:19:27.380471 393 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.055s user 0.005s sys 0.158s
I20260618 22:19:27.380606 393 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260618 22:19:27.380733 393 mini_ranger_kms.cc:401] 127.0.98.84:40803/kms/v1/keys
Jun 18 22:19:27 dist-test-slave-g6r4 krb5kdc[11163](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781821167, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 18 22:19:28 dist-test-slave-g6r4 krb5kdc[11163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821167, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.84@KRBTEST.COM for HTTP/127.0.98.84@KRBTEST.COM
Jun 18 22:19:28 dist-test-slave-g6r4 krb5kdc[11163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821167, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.84@KRBTEST.COM for HTTP/127.0.98.84@KRBTEST.COM
Jun 18 22:19:30 dist-test-slave-g6r4 krb5kdc[11163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821170, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260618 22:19:30.607138 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:33901
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:39991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:33901
--ranger_config_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.84:40803/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:19:30.720103 11607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:19:30.720357 11607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:19:30.720395 11607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:19:30.724927 11607 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:19:30.724987 11607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:19:30.725008 11607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:19:30.725026 11607 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:19:30.725042 11607 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:19:30.730648 11607 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:39991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.84:40803/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:33901
--ranger_config_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:33901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11607
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:19:30.731688 11607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:19:30.732825 11607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:19:30.742345 11613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:19:30.742326 11612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:19:30.742439 11607 server_base.cc:1034] running on GCE node
W20260618 22:19:30.742317 11615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:19:30.743168 11607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:19:30.744270 11607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:19:30.745458 11607 hybrid_clock.cc:648] HybridClock initialized: now 1781821170745439 us; error 38 us; skew 500 ppm
Jun 18 22:19:30 dist-test-slave-g6r4 krb5kdc[11163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821170, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:19:30.748365 11607 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260618 22:19:30.749297 11607 webserver.cc:460] Webserver started at http://127.0.98.126:33111/ using document root <none> and password file <none>
I20260618 22:19:30.749819 11607 fs_manager.cc:359] Metadata directory not provided
I20260618 22:19:30.749894 11607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:19:30.750115 11607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 18 22:19:30 dist-test-slave-g6r4 krb5kdc[11163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821170, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for HTTP/127.0.98.84@KRBTEST.COM
I20260618 22:19:30.810667 11607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/master-0/data/instance:
uuid: "3db860900e304417be41cb4118793d8a"
format_stamp: "Formatted at 2026-06-18 22:19:30 on dist-test-slave-g6r4"
server_key: "5bdcb0d539bd313dc16286b3acdc869d"
server_key_iv: "7ad03571019a30bf85033fede28cdbe0"
server_key_version: "kuduclusterkey@0"
I20260618 22:19:30.811296 11607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/master-0/wal/instance:
uuid: "3db860900e304417be41cb4118793d8a"
format_stamp: "Formatted at 2026-06-18 22:19:30 on dist-test-slave-g6r4"
server_key: "5bdcb0d539bd313dc16286b3acdc869d"
server_key_iv: "7ad03571019a30bf85033fede28cdbe0"
server_key_version: "kuduclusterkey@0"
I20260618 22:19:30.815269 11607 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260618 22:19:30.833005 11622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:19:30.834707 11607 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:19:30.834846 11607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/master-0/wal
uuid: "3db860900e304417be41cb4118793d8a"
format_stamp: "Formatted at 2026-06-18 22:19:30 on dist-test-slave-g6r4"
server_key: "5bdcb0d539bd313dc16286b3acdc869d"
server_key_iv: "7ad03571019a30bf85033fede28cdbe0"
server_key_version: "kuduclusterkey@0"
I20260618 22:19:30.834975 11607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:19:30.852012 11607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:19:30.854928 11607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:19:30.855083 11607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:19:30.862313 11607 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:33901
I20260618 22:19:30.862340 11674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:33901 every 8 connection(s)
I20260618 22:19:30.863201 11607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
I20260618 22:19:30.864415 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 11607
I20260618 22:19:30.864545 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/master-0/wal/instance
I20260618 22:19:30.864867 393 external_mini_cluster.cc:1428] Setting key 71f69aff13971b17eb48ac9986f6acb7
I20260618 22:19:30.866240 11675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:19:30.872840 11675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3db860900e304417be41cb4118793d8a: Bootstrap starting.
Jun 18 22:19:30 dist-test-slave-g6r4 krb5kdc[11163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821170, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:19:30.875216 11675 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3db860900e304417be41cb4118793d8a: Neither blocks nor log segments found. Creating new log.
I20260618 22:19:30.876579 11675 log.cc:824] T 00000000000000000000000000000000 P 3db860900e304417be41cb4118793d8a: Log is configured to *not* fsync() on all Append() calls
I20260618 22:19:30.878733 11675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3db860900e304417be41cb4118793d8a: No bootstrap required, opened a new log
I20260618 22:19:30.879987 11678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:19:30.866259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39868 (local address 127.0.98.126:33901)
0618 22:19:30.866675 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0618 22:19:30.866702 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0618 22:19:30.866743 (+ 41us) server_negotiation.cc:407] Connection header received
0618 22:19:30.867538 (+ 795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:19:30.867559 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:19:30.867960 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:19:30.868309 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:19:30.868955 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:19:30.870426 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:19:30.871298 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:19:30.872006 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:19:30.872161 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:19:30.875347 (+ 3186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:19:30.875355 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:19:30.875370 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:19:30.875404 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:19:30.877196 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:19:30.877603 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:19:30.877608 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:19:30.877614 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:19:30.877742 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:19:30.878028 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:19:30.878031 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:19:30.878032 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:19:30.878480 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:19:30.878664 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0618 22:19:30.878885 (+ 221us) server_negotiation.cc:299] Negotiation successful
0618 22:19:30.879145 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":116,"threads_started":1}
I20260618 22:19:30.882210 11675 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3db860900e304417be41cb4118793d8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3db860900e304417be41cb4118793d8a" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33901 } }
I20260618 22:19:30.882401 11675 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3db860900e304417be41cb4118793d8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:19:30.882434 11675 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3db860900e304417be41cb4118793d8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3db860900e304417be41cb4118793d8a, State: Initialized, Role: FOLLOWER
I20260618 22:19:30.882761 11675 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3db860900e304417be41cb4118793d8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3db860900e304417be41cb4118793d8a" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33901 } }
I20260618 22:19:30.882861 11675 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3db860900e304417be41cb4118793d8a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:19:30.882905 11675 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3db860900e304417be41cb4118793d8a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:19:30.882970 11675 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3db860900e304417be41cb4118793d8a [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:19:30.883757 11675 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3db860900e304417be41cb4118793d8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3db860900e304417be41cb4118793d8a" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33901 } }
I20260618 22:19:30.884029 11675 leader_election.cc:302] T 00000000000000000000000000000000 P 3db860900e304417be41cb4118793d8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3db860900e304417be41cb4118793d8a; no voters:
I20260618 22:19:30.884264 11675 leader_election.cc:288] T 00000000000000000000000000000000 P 3db860900e304417be41cb4118793d8a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:19:30.884374 11680 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3db860900e304417be41cb4118793d8a [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:19:30.884594 11680 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3db860900e304417be41cb4118793d8a [term 1 LEADER]: Becoming Leader. State: Replica: 3db860900e304417be41cb4118793d8a, State: Running, Role: LEADER
I20260618 22:19:30.884912 11680 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3db860900e304417be41cb4118793d8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3db860900e304417be41cb4118793d8a" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33901 } }
I20260618 22:19:30.885445 11675 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3db860900e304417be41cb4118793d8a [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:19:30.886606 11682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3db860900e304417be41cb4118793d8a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3db860900e304417be41cb4118793d8a. Latest consensus state: current_term: 1 leader_uuid: "3db860900e304417be41cb4118793d8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3db860900e304417be41cb4118793d8a" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33901 } } }
I20260618 22:19:30.886641 11681 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3db860900e304417be41cb4118793d8a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3db860900e304417be41cb4118793d8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3db860900e304417be41cb4118793d8a" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 33901 } } }
I20260618 22:19:30.886751 11681 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3db860900e304417be41cb4118793d8a [sys.catalog]: This master's current role is: LEADER
I20260618 22:19:30.886750 11682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3db860900e304417be41cb4118793d8a [sys.catalog]: This master's current role is: LEADER
I20260618 22:19:30.887749 11675 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260618 22:19:30.890452 11689 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:19:30.895414 11689 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:19:30.905638 11689 catalog_manager.cc:1347] Generated new cluster ID: 3824a95718e74211b8b018d4bc85ab5a
I20260618 22:19:30.905730 11689 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:19:30.960744 11689 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:19:30.961967 11689 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:19:30.969129 11689 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3db860900e304417be41cb4118793d8a: Generated new TSK 0
I20260618 22:19:30.970089 11689 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab.
I20260618 22:19:31.662189 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.84:40803/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:33901
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:39991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:19:31.779628 11720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:19:31.779873 11720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:19:31.779915 11720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:19:31.784183 11720 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:19:31.784235 11720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:19:31.784320 11720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260618 22:19:31.790556 11720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:39991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.84:40803/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:33901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11720
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:19:31.791942 11720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:19:31.793241 11720 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:19:31.803910 11728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:19:31.803901 11727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:19:31.803901 11730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:19:31.804117 11720 server_base.cc:1034] running on GCE node
I20260618 22:19:31.804697 11720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:19:31.805365 11720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:19:31.806542 11720 hybrid_clock.cc:648] HybridClock initialized: now 1781821171806517 us; error 45 us; skew 500 ppm
Jun 18 22:19:31 dist-test-slave-g6r4 krb5kdc[11163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:19:31.809755 11720 init.cc:377] Logged in from keytab as kudu/127.0.98.65@KRBTEST.COM (short username kudu)
I20260618 22:19:31.810837 11720 webserver.cc:460] Webserver started at http://127.0.98.65:38055/ using document root <none> and password file <none>
I20260618 22:19:31.811549 11720 fs_manager.cc:359] Metadata directory not provided
I20260618 22:19:31.811640 11720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:19:31.811928 11720 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 18 22:19:31 dist-test-slave-g6r4 krb5kdc[11163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for HTTP/127.0.98.84@KRBTEST.COM
I20260618 22:19:31.832831 11720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/ts-0/data/instance:
uuid: "3a6697f25ce748709f4d411ab4e01081"
format_stamp: "Formatted at 2026-06-18 22:19:31 on dist-test-slave-g6r4"
server_key: "4a5fbcf82b4c2e24116f0f5314516d64"
server_key_iv: "8cbdbc223b7416a62b78e2e9d646e618"
server_key_version: "kuduclusterkey@0"
I20260618 22:19:31.835614 11720 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/ts-0/wal/instance:
uuid: "3a6697f25ce748709f4d411ab4e01081"
format_stamp: "Formatted at 2026-06-18 22:19:31 on dist-test-slave-g6r4"
server_key: "4a5fbcf82b4c2e24116f0f5314516d64"
server_key_iv: "8cbdbc223b7416a62b78e2e9d646e618"
server_key_version: "kuduclusterkey@0"
I20260618 22:19:31.843258 11720 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.000s
I20260618 22:19:31.859112 11737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:19:31.860698 11720 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260618 22:19:31.860814 11720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/ts-0/wal
uuid: "3a6697f25ce748709f4d411ab4e01081"
format_stamp: "Formatted at 2026-06-18 22:19:31 on dist-test-slave-g6r4"
server_key: "4a5fbcf82b4c2e24116f0f5314516d64"
server_key_iv: "8cbdbc223b7416a62b78e2e9d646e618"
server_key_version: "kuduclusterkey@0"
I20260618 22:19:31.860929 11720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:19:31.875981 11720 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:19:31.879546 11720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:19:31.879750 11720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:19:31.880348 11720 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:19:31.881251 11720 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:19:31.881330 11720 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:19:31.881418 11720 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:19:31.881460 11720 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:19:31.893656 11720 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:33073
I20260618 22:19:31.895054 11720 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
I20260618 22:19:31.897172 11850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:33073 every 8 connection(s)
I20260618 22:19:31.899637 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 11720
I20260618 22:19:31.899770 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/ts-0/wal/instance
I20260618 22:19:31.900141 393 external_mini_cluster.cc:1428] Setting key 607596d20166040e3b4525793e7b474e
Jun 18 22:19:31 dist-test-slave-g6r4 krb5kdc[11163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.66@KRBTEST.COM" created.
I20260618 22:19:31.920994 11853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:19:31.899571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:34119 (local address 127.0.98.126:33901)
0618 22:19:31.904917 (+ 5346us) server_negotiation.cc:206] Beginning negotiation
0618 22:19:31.904927 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0618 22:19:31.906622 (+ 1695us) server_negotiation.cc:407] Connection header received
0618 22:19:31.906670 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:19:31.906673 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:19:31.906730 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:19:31.906821 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:19:31.908391 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:19:31.909278 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:19:31.910738 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:19:31.911194 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:19:31.911439 (+ 245us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:19:31.914336 (+ 2897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:19:31.914340 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:19:31.914344 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:19:31.914371 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:19:31.916328 (+ 1957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:19:31.917065 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:19:31.917069 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:19:31.917071 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:19:31.917163 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:19:31.919540 (+ 2377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:19:31.919544 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:19:31.919546 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:19:31.919810 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:19:31.919913 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0618 22:19:31.920631 (+ 718us) server_negotiation.cc:299] Negotiation successful
0618 22:19:31.920786 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5259,"thread_start_us":118,"threads_started":1}
I20260618 22:19:31.922883 11854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:19:31.900057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:33901 (local address 127.0.98.65:34119)
0618 22:19:31.905211 (+ 5154us) negotiation.cc:107] Waiting for socket to connect
0618 22:19:31.905280 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0618 22:19:31.906323 (+ 1043us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:19:31.907143 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:19:31.907152 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:19:31.907505 (+ 353us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:19:31.908203 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:19:31.908224 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:19:31.909450 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:19:31.909456 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:19:31.910599 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:19:31.910608 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:19:31.911421 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:19:31.911427 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:19:31.911652 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:19:31.912397 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:19:31.912421 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:19:31.914137 (+ 1716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:19:31.916456 (+ 2319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:19:31.916463 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:19:31.916478 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:19:31.916927 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:19:31.917273 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:19:31.917277 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:19:31.917279 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:19:31.919393 (+ 2114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:19:31.919989 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:19:31.919996 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:19:31.920333 (+ 337us) client_negotiation.cc:769] Sending connection context
0618 22:19:31.921469 (+ 1136us) client_negotiation.cc:240] Negotiation successful
0618 22:19:31.921821 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4924,"thread_start_us":154,"threads_started":1}
I20260618 22:19:31.924607 11851 heartbeater.cc:347] Connected to a master server at 127.0.98.126:33901
I20260618 22:19:31.924868 11851 heartbeater.cc:464] Registering TS with master...
I20260618 22:19:31.925438 11851 heartbeater.cc:511] Master 127.0.98.126:33901 requested a full tablet report, sending...
I20260618 22:19:31.927301 11639 ts_manager.cc:194] Registered new tserver with Master: 3a6697f25ce748709f4d411ab4e01081 (127.0.98.65:33073)
I20260618 22:19:31.928900 11639 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.65@KRBTEST.COM'} at 127.0.98.65:34119
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab.
I20260618 22:19:31.934888 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.84:40803/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:33901
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:39991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 18 22:19:31 dist-test-slave-g6r4 krb5kdc[11163](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781821171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260618 22:19:32.056600 11859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:19:32.056842 11859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:19:32.056883 11859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:19:32.061200 11859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:19:32.061277 11859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:19:32.061380 11859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260618 22:19:32.067602 11859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:39991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.84:40803/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:33901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:19:32.068856 11859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:19:32.069851 11859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:19:32.079437 11865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:19:32.079634 11866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:19:32.079742 11868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:19:32.079973 11859 server_base.cc:1034] running on GCE node
I20260618 22:19:32.080585 11859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:19:32.081306 11859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:19:32.082479 11859 hybrid_clock.cc:648] HybridClock initialized: now 1781821172082414 us; error 82 us; skew 500 ppm
Jun 18 22:19:32 dist-test-slave-g6r4 krb5kdc[11163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:19:32.085860 11859 init.cc:377] Logged in from keytab as kudu/127.0.98.66@KRBTEST.COM (short username kudu)
I20260618 22:19:32.086910 11859 webserver.cc:460] Webserver started at http://127.0.98.66:45413/ using document root <none> and password file <none>
I20260618 22:19:32.087574 11859 fs_manager.cc:359] Metadata directory not provided
I20260618 22:19:32.087657 11859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:19:32.087904 11859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 18 22:19:32 dist-test-slave-g6r4 krb5kdc[11163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for HTTP/127.0.98.84@KRBTEST.COM
I20260618 22:19:32.103026 11859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/ts-1/data/instance:
uuid: "702a429ede7b4d2a9f6b2789bb78655c"
format_stamp: "Formatted at 2026-06-18 22:19:32 on dist-test-slave-g6r4"
server_key: "7374807c0bb8a8b7bc3deb47065e1011"
server_key_iv: "4bfe391684c14d024577b35d8b5447f9"
server_key_version: "kuduclusterkey@0"
I20260618 22:19:32.103724 11859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/ts-1/wal/instance:
uuid: "702a429ede7b4d2a9f6b2789bb78655c"
format_stamp: "Formatted at 2026-06-18 22:19:32 on dist-test-slave-g6r4"
server_key: "7374807c0bb8a8b7bc3deb47065e1011"
server_key_iv: "4bfe391684c14d024577b35d8b5447f9"
server_key_version: "kuduclusterkey@0"
I20260618 22:19:32.109686 11859 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260618 22:19:32.134354 11875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:19:32.136011 11859 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.002s sys 0.001s
I20260618 22:19:32.136178 11859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/ts-1/wal
uuid: "702a429ede7b4d2a9f6b2789bb78655c"
format_stamp: "Formatted at 2026-06-18 22:19:32 on dist-test-slave-g6r4"
server_key: "7374807c0bb8a8b7bc3deb47065e1011"
server_key_iv: "4bfe391684c14d024577b35d8b5447f9"
server_key_version: "kuduclusterkey@0"
I20260618 22:19:32.136353 11859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:19:32.166887 11859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:19:32.170665 11859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:19:32.170945 11859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:19:32.171677 11859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:19:32.172819 11859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:19:32.172909 11859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:19:32.172999 11859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:19:32.173056 11859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:19:32.183511 11859 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:33647
I20260618 22:19:32.184645 11859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
I20260618 22:19:32.192818 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 11859
I20260618 22:19:32.192939 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/ts-1/wal/instance
I20260618 22:19:32.193224 393 external_mini_cluster.cc:1428] Setting key 595eaa562192829d9617c16d2c743a3b
I20260618 22:19:32.196863 11988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:33647 every 8 connection(s)
WARNING: no policy specified for kudu/127.0.98.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.67@KRBTEST.COM" created.
Jun 18 22:19:32 dist-test-slave-g6r4 krb5kdc[11163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:19:32.226176 11853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:19:32.199343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:58465 (local address 127.0.98.126:33901)
0618 22:19:32.199476 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0618 22:19:32.199480 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:19:32.206920 (+ 7440us) server_negotiation.cc:407] Connection header received
0618 22:19:32.208443 (+ 1523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:19:32.208482 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:19:32.208540 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:19:32.208602 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:19:32.212351 (+ 3749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:19:32.213198 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:19:32.214937 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:19:32.215280 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:19:32.215359 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:19:32.218422 (+ 3063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:19:32.218426 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:19:32.218429 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:19:32.218475 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:19:32.220387 (+ 1912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:19:32.221021 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:19:32.221025 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:19:32.221026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:19:32.221118 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:19:32.221538 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:19:32.221541 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:19:32.221543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:19:32.221785 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:19:32.221869 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0618 22:19:32.225411 (+ 3542us) server_negotiation.cc:299] Negotiation successful
0618 22:19:32.225557 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260618 22:19:32.228099 11992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:19:32.198986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:33901 (local address 127.0.98.66:58465)
0618 22:19:32.206730 (+ 7744us) negotiation.cc:107] Waiting for socket to connect
0618 22:19:32.206808 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0618 22:19:32.208170 (+ 1362us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:19:32.211077 (+ 2907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:19:32.211088 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:19:32.211436 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:19:32.212165 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:19:32.212187 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:19:32.213370 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:19:32.213374 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:19:32.214788 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:19:32.214801 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:19:32.215566 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:19:32.215570 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:19:32.215789 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:19:32.216480 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:19:32.216501 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:19:32.218207 (+ 1706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:19:32.220524 (+ 2317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:19:32.220530 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:19:32.220546 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:19:32.220897 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:19:32.221230 (+ 333us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:19:32.221233 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:19:32.221236 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:19:32.221419 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:19:32.221959 (+ 540us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:19:32.221965 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:19:32.222286 (+ 321us) client_negotiation.cc:769] Sending connection context
0618 22:19:32.226650 (+ 4364us) client_negotiation.cc:240] Negotiation successful
0618 22:19:32.227014 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7484,"thread_start_us":165,"threads_started":1}
I20260618 22:19:32.229671 11989 heartbeater.cc:347] Connected to a master server at 127.0.98.126:33901
I20260618 22:19:32.229955 11989 heartbeater.cc:464] Registering TS with master...
I20260618 22:19:32.230511 11989 heartbeater.cc:511] Master 127.0.98.126:33901 requested a full tablet report, sending...
I20260618 22:19:32.231635 11639 ts_manager.cc:194] Registered new tserver with Master: 702a429ede7b4d2a9f6b2789bb78655c (127.0.98.66:33647)
I20260618 22:19:32.232390 11639 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.66@KRBTEST.COM'} at 127.0.98.66:58465
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab.
I20260618 22:19:32.254348 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.84:40803/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:33901
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:39991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:19:32.403239 11994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:19:32.403671 11994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:19:32.403816 11994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:19:32.410077 11994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:19:32.410231 11994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:19:32.410383 11994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260618 22:19:32.418507 11994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:39991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.84:40803/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:33901
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11994
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:19:32.420184 11994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:19:32.421598 11994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:19:32.435335 12000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:19:32.436365 11999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:19:32.436699 11994 server_base.cc:1034] running on GCE node
W20260618 22:19:32.437125 12002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:19:32.437661 11994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:19:32.438473 11994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:19:32.439704 11994 hybrid_clock.cc:648] HybridClock initialized: now 1781821172439643 us; error 78 us; skew 500 ppm
Jun 18 22:19:32 dist-test-slave-g6r4 krb5kdc[11163](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:19:32.443706 11994 init.cc:377] Logged in from keytab as kudu/127.0.98.67@KRBTEST.COM (short username kudu)
I20260618 22:19:32.444828 11994 webserver.cc:460] Webserver started at http://127.0.98.67:35771/ using document root <none> and password file <none>
I20260618 22:19:32.445518 11994 fs_manager.cc:359] Metadata directory not provided
I20260618 22:19:32.445597 11994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:19:32.445847 11994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 18 22:19:32 dist-test-slave-g6r4 krb5kdc[11163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for HTTP/127.0.98.84@KRBTEST.COM
I20260618 22:19:32.467468 11994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/ts-2/data/instance:
uuid: "7189161cfb724373947bc63a513b7656"
format_stamp: "Formatted at 2026-06-18 22:19:32 on dist-test-slave-g6r4"
server_key: "019751a1a07f5c811e5eb243993006be"
server_key_iv: "1158a212f06bbbb38ef7a118f70a4102"
server_key_version: "kuduclusterkey@0"
I20260618 22:19:32.468519 11994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/ts-2/wal/instance:
uuid: "7189161cfb724373947bc63a513b7656"
format_stamp: "Formatted at 2026-06-18 22:19:32 on dist-test-slave-g6r4"
server_key: "019751a1a07f5c811e5eb243993006be"
server_key_iv: "1158a212f06bbbb38ef7a118f70a4102"
server_key_version: "kuduclusterkey@0"
I20260618 22:19:32.473826 11994 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260618 22:19:32.495766 12009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:19:32.497452 11994 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260618 22:19:32.497570 11994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/ts-2/wal
uuid: "7189161cfb724373947bc63a513b7656"
format_stamp: "Formatted at 2026-06-18 22:19:32 on dist-test-slave-g6r4"
server_key: "019751a1a07f5c811e5eb243993006be"
server_key_iv: "1158a212f06bbbb38ef7a118f70a4102"
server_key_version: "kuduclusterkey@0"
I20260618 22:19:32.497690 11994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:19:32.514940 11994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:19:32.518688 11994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:19:32.519018 11994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:19:32.519747 11994 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:19:32.520821 11994 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:19:32.520941 11994 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:19:32.521050 11994 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:19:32.521122 11994 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:19:32.535502 11994 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:43215
I20260618 22:19:32.536691 11994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
I20260618 22:19:32.537684 12122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:43215 every 8 connection(s)
I20260618 22:19:32.538487 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 11994
I20260618 22:19:32.538605 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/ts-2/wal/instance
I20260618 22:19:32.538903 393 external_mini_cluster.cc:1428] Setting key 2bbd7b8b8a5576ab34749869b31a2c94
Jun 18 22:19:32 dist-test-slave-g6r4 krb5kdc[11163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821172, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
Jun 18 22:19:32 dist-test-slave-g6r4 krb5kdc[11163](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for HTTP/127.0.98.84@KRBTEST.COM
I20260618 22:19:32.556335 11853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:19:32.541532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:50219 (local address 127.0.98.126:33901)
0618 22:19:32.541645 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0618 22:19:32.541650 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:19:32.542225 (+ 575us) server_negotiation.cc:407] Connection header received
0618 22:19:32.544170 (+ 1945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:19:32.544174 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:19:32.544229 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:19:32.544285 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:19:32.545944 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:19:32.546757 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:19:32.548442 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:19:32.548793 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:19:32.548895 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:19:32.551905 (+ 3010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:19:32.551909 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:19:32.551912 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:19:32.551937 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:19:32.553726 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:19:32.554525 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:19:32.554529 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:19:32.554531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:19:32.554628 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:19:32.555052 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:19:32.555055 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:19:32.555058 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:19:32.555287 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:19:32.555378 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0618 22:19:32.556048 (+ 670us) server_negotiation.cc:299] Negotiation successful
0618 22:19:32.556172 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260618 22:19:32.558362 12125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:19:32.540491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:33901 (local address 127.0.98.67:50219)
0618 22:19:32.542084 (+ 1593us) negotiation.cc:107] Waiting for socket to connect
0618 22:19:32.542139 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0618 22:19:32.543697 (+ 1558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:19:32.544691 (+ 994us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:19:32.544713 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:19:32.545086 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:19:32.545746 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:19:32.545769 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:19:32.546917 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:19:32.546920 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:19:32.548300 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:19:32.548313 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:19:32.549093 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:19:32.549097 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:19:32.549296 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:19:32.549980 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:19:32.550001 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:19:32.551734 (+ 1733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:19:32.554035 (+ 2301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:19:32.554042 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:19:32.554058 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:19:32.554397 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:19:32.554760 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:19:32.554763 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:19:32.554765 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:19:32.554937 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:19:32.555444 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:19:32.555451 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:19:32.555776 (+ 325us) client_negotiation.cc:769] Sending connection context
0618 22:19:32.556948 (+ 1172us) client_negotiation.cc:240] Negotiation successful
0618 22:19:32.557287 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1386,"thread_start_us":129,"threads_started":1}
I20260618 22:19:32.559792 12123 heartbeater.cc:347] Connected to a master server at 127.0.98.126:33901
I20260618 22:19:32.560014 12123 heartbeater.cc:464] Registering TS with master...
I20260618 22:19:32.560524 12123 heartbeater.cc:511] Master 127.0.98.126:33901 requested a full tablet report, sending...
I20260618 22:19:32.561626 11639 ts_manager.cc:194] Registered new tserver with Master: 7189161cfb724373947bc63a513b7656 (127.0.98.67:43215)
I20260618 22:19:32.562376 11639 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.67@KRBTEST.COM'} at 127.0.98.67:50219
I20260618 22:19:32.565606 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 22:19:32.600970 11853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:19:32.577868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39880 (local address 127.0.98.126:33901)
0618 22:19:32.578018 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0618 22:19:32.578022 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:19:32.582808 (+ 4786us) server_negotiation.cc:407] Connection header received
0618 22:19:32.583014 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:19:32.583018 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:19:32.583067 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:19:32.583120 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:19:32.584878 (+ 1758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:19:32.585704 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:19:32.587274 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:19:32.587632 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:19:32.587719 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:19:32.595406 (+ 7687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:19:32.595410 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:19:32.595413 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:19:32.595441 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:19:32.597195 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:19:32.598503 (+ 1308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:19:32.598507 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:19:32.598509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:19:32.598597 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:19:32.599332 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:19:32.599339 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:19:32.599342 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:19:32.599634 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:19:32.599782 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0618 22:19:32.600675 (+ 893us) server_negotiation.cc:299] Negotiation successful
0618 22:19:32.600802 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260618 22:19:32.605067 11639 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39880:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 22:19:32.609191 11639 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:19:32.677294 12136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:19:32.662325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:33073 (local address 127.0.0.1:48132)
0618 22:19:32.669643 (+ 7318us) negotiation.cc:107] Waiting for socket to connect
0618 22:19:32.669706 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0618 22:19:32.669840 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:19:32.671466 (+ 1626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:19:32.671472 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:19:32.671496 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:19:32.671636 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:19:32.671644 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:19:32.673066 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:19:32.673070 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:19:32.674489 (+ 1419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:19:32.674500 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:19:32.676704 (+ 2204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:19:32.676711 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:19:32.676832 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:19:32.676895 (+ 63us) client_negotiation.cc:769] Sending connection context
0618 22:19:32.677003 (+ 108us) client_negotiation.cc:240] Negotiation successful
0618 22:19:32.677124 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7119,"thread_start_us":104,"threads_started":1}
I20260618 22:19:32.679554 12140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:19:32.664223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48132 (local address 127.0.98.65:33073)
0618 22:19:32.670416 (+ 6193us) server_negotiation.cc:206] Beginning negotiation
0618 22:19:32.670424 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0618 22:19:32.670449 (+ 25us) server_negotiation.cc:407] Connection header received
0618 22:19:32.670842 (+ 393us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:19:32.670850 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:19:32.671098 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:19:32.671278 (+ 180us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:19:32.671774 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:19:32.672944 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:19:32.674637 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:19:32.675477 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:19:32.675581 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:19:32.675749 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0618 22:19:32.679309 (+ 3560us) server_negotiation.cc:299] Negotiation successful
0618 22:19:32.679418 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6084,"thread_start_us":59,"threads_started":1}
I20260618 22:19:32.680110 12135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:19:32.663635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:33647 (local address 127.0.0.1:57776)
0618 22:19:32.670590 (+ 6955us) negotiation.cc:107] Waiting for socket to connect
0618 22:19:32.670602 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 22:19:32.670676 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:19:32.674807 (+ 4131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:19:32.674811 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:19:32.674832 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:19:32.674921 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:19:32.674927 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:19:32.676426 (+ 1499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:19:32.676430 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:19:32.677634 (+ 1204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:19:32.677645 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:19:32.679721 (+ 2076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:19:32.679725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:19:32.679815 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:19:32.679839 (+ 24us) client_negotiation.cc:769] Sending connection context
0618 22:19:32.679911 (+ 72us) client_negotiation.cc:240] Negotiation successful
0618 22:19:32.679975 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6900,"thread_start_us":91,"threads_started":1}
I20260618 22:19:32.680606 12139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:19:32.663847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57776 (local address 127.0.98.66:33647)
0618 22:19:32.674115 (+ 10268us) server_negotiation.cc:206] Beginning negotiation
0618 22:19:32.674134 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0618 22:19:32.674160 (+ 26us) server_negotiation.cc:407] Connection header received
0618 22:19:32.674235 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:19:32.674240 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:19:32.674455 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:19:32.674638 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:19:32.675058 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:19:32.676283 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:19:32.678588 (+ 2305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:19:32.679396 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:19:32.679503 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:19:32.679643 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0618 22:19:32.680374 (+ 731us) server_negotiation.cc:299] Negotiation successful
0618 22:19:32.680472 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10142,"thread_start_us":79,"threads_started":1}
I20260618 22:19:32.683799 11915 tablet_service.cc:1467] Processing CreateTablet for tablet 8edb24e97d314dc394ac680bc8112117 (DEFAULT_TABLE table=test-table [id=a649271a8ec5421d8105396916ae7398]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:19:32.684965 11915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8edb24e97d314dc394ac680bc8112117. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:19:32.684931 11770 tablet_service.cc:1467] Processing CreateTablet for tablet 8edb24e97d314dc394ac680bc8112117 (DEFAULT_TABLE table=test-table [id=a649271a8ec5421d8105396916ae7398]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:19:32.690040 11770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8edb24e97d314dc394ac680bc8112117. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:19:32.697881 12138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:19:32.663269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:43215 (local address 127.0.0.1:49770)
0618 22:19:32.670063 (+ 6794us) negotiation.cc:107] Waiting for socket to connect
0618 22:19:32.670076 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:19:32.670160 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:19:32.678423 (+ 8263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:19:32.678427 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:19:32.678445 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:19:32.678539 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:19:32.678545 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:19:32.694944 (+ 16399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:19:32.694948 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:19:32.696100 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:19:32.696113 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:19:32.697464 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:19:32.697468 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:19:32.697564 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:19:32.697587 (+ 23us) client_negotiation.cc:769] Sending connection context
0618 22:19:32.697655 (+ 68us) client_negotiation.cc:240] Negotiation successful
0618 22:19:32.697726 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6733,"thread_start_us":108,"threads_started":1}
I20260618 22:19:32.698452 12137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:19:32.663452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49770 (local address 127.0.98.67:43215)
0618 22:19:32.677967 (+ 14515us) server_negotiation.cc:206] Beginning negotiation
0618 22:19:32.677980 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0618 22:19:32.678004 (+ 24us) server_negotiation.cc:407] Connection header received
0618 22:19:32.678077 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:19:32.678082 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:19:32.678277 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:19:32.678426 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:19:32.678921 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:19:32.694785 (+ 15864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:19:32.696295 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:19:32.697160 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:19:32.697268 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:19:32.697405 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0618 22:19:32.698162 (+ 757us) server_negotiation.cc:299] Negotiation successful
0618 22:19:32.698315 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14384,"thread_start_us":124,"threads_started":1}
I20260618 22:19:32.701417 12057 tablet_service.cc:1467] Processing CreateTablet for tablet 8edb24e97d314dc394ac680bc8112117 (DEFAULT_TABLE table=test-table [id=a649271a8ec5421d8105396916ae7398]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:19:32.702653 12057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8edb24e97d314dc394ac680bc8112117. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:19:32.710260 12142 tablet_bootstrap.cc:492] T 8edb24e97d314dc394ac680bc8112117 P 3a6697f25ce748709f4d411ab4e01081: Bootstrap starting.
I20260618 22:19:32.711055 12141 tablet_bootstrap.cc:492] T 8edb24e97d314dc394ac680bc8112117 P 702a429ede7b4d2a9f6b2789bb78655c: Bootstrap starting.
I20260618 22:19:32.713117 12142 tablet_bootstrap.cc:654] T 8edb24e97d314dc394ac680bc8112117 P 3a6697f25ce748709f4d411ab4e01081: Neither blocks nor log segments found. Creating new log.
I20260618 22:19:32.714148 12142 log.cc:824] T 8edb24e97d314dc394ac680bc8112117 P 3a6697f25ce748709f4d411ab4e01081: Log is configured to *not* fsync() on all Append() calls
I20260618 22:19:32.714262 12141 tablet_bootstrap.cc:654] T 8edb24e97d314dc394ac680bc8112117 P 702a429ede7b4d2a9f6b2789bb78655c: Neither blocks nor log segments found. Creating new log.
I20260618 22:19:32.718240 12143 tablet_bootstrap.cc:492] T 8edb24e97d314dc394ac680bc8112117 P 7189161cfb724373947bc63a513b7656: Bootstrap starting.
I20260618 22:19:32.720795 12141 log.cc:824] T 8edb24e97d314dc394ac680bc8112117 P 702a429ede7b4d2a9f6b2789bb78655c: Log is configured to *not* fsync() on all Append() calls
I20260618 22:19:32.721880 12143 tablet_bootstrap.cc:654] T 8edb24e97d314dc394ac680bc8112117 P 7189161cfb724373947bc63a513b7656: Neither blocks nor log segments found. Creating new log.
I20260618 22:19:32.723097 12143 log.cc:824] T 8edb24e97d314dc394ac680bc8112117 P 7189161cfb724373947bc63a513b7656: Log is configured to *not* fsync() on all Append() calls
I20260618 22:19:32.727716 12142 tablet_bootstrap.cc:492] T 8edb24e97d314dc394ac680bc8112117 P 3a6697f25ce748709f4d411ab4e01081: No bootstrap required, opened a new log
I20260618 22:19:32.727968 12142 ts_tablet_manager.cc:1397] T 8edb24e97d314dc394ac680bc8112117 P 3a6697f25ce748709f4d411ab4e01081: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.000s
I20260618 22:19:32.732095 12141 tablet_bootstrap.cc:492] T 8edb24e97d314dc394ac680bc8112117 P 702a429ede7b4d2a9f6b2789bb78655c: No bootstrap required, opened a new log
I20260618 22:19:32.732389 12141 ts_tablet_manager.cc:1397] T 8edb24e97d314dc394ac680bc8112117 P 702a429ede7b4d2a9f6b2789bb78655c: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.000s
I20260618 22:19:32.732374 12142 raft_consensus.cc:348] T 8edb24e97d314dc394ac680bc8112117 P 3a6697f25ce748709f4d411ab4e01081 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a6697f25ce748709f4d411ab4e01081" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33073 } } peers { permanent_uuid: "7189161cfb724373947bc63a513b7656" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43215 } } peers { permanent_uuid: "702a429ede7b4d2a9f6b2789bb78655c" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 33647 } }
I20260618 22:19:32.732761 12142 raft_consensus.cc:374] T 8edb24e97d314dc394ac680bc8112117 P 3a6697f25ce748709f4d411ab4e01081 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:19:32.732857 12142 raft_consensus.cc:729] T 8edb24e97d314dc394ac680bc8112117 P 3a6697f25ce748709f4d411ab4e01081 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a6697f25ce748709f4d411ab4e01081, State: Initialized, Role: FOLLOWER
I20260618 22:19:32.733385 12142 consensus_queue.cc:260] T 8edb24e97d314dc394ac680bc8112117 P 3a6697f25ce748709f4d411ab4e01081 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a6697f25ce748709f4d411ab4e01081" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33073 } } peers { permanent_uuid: "7189161cfb724373947bc63a513b7656" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43215 } } peers { permanent_uuid: "702a429ede7b4d2a9f6b2789bb78655c" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 33647 } }
I20260618 22:19:32.735265 12142 ts_tablet_manager.cc:1428] T 8edb24e97d314dc394ac680bc8112117 P 3a6697f25ce748709f4d411ab4e01081: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260618 22:19:32.735719 11851 heartbeater.cc:503] Master 127.0.98.126:33901 was elected leader, sending a full tablet report...
I20260618 22:19:32.736738 12141 raft_consensus.cc:348] T 8edb24e97d314dc394ac680bc8112117 P 702a429ede7b4d2a9f6b2789bb78655c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a6697f25ce748709f4d411ab4e01081" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33073 } } peers { permanent_uuid: "7189161cfb724373947bc63a513b7656" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43215 } } peers { permanent_uuid: "702a429ede7b4d2a9f6b2789bb78655c" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 33647 } }
I20260618 22:19:32.737017 12141 raft_consensus.cc:374] T 8edb24e97d314dc394ac680bc8112117 P 702a429ede7b4d2a9f6b2789bb78655c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:19:32.737073 12141 raft_consensus.cc:729] T 8edb24e97d314dc394ac680bc8112117 P 702a429ede7b4d2a9f6b2789bb78655c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 702a429ede7b4d2a9f6b2789bb78655c, State: Initialized, Role: FOLLOWER
I20260618 22:19:32.739323 12143 tablet_bootstrap.cc:492] T 8edb24e97d314dc394ac680bc8112117 P 7189161cfb724373947bc63a513b7656: No bootstrap required, opened a new log
I20260618 22:19:32.739521 12143 ts_tablet_manager.cc:1397] T 8edb24e97d314dc394ac680bc8112117 P 7189161cfb724373947bc63a513b7656: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.003s
I20260618 22:19:32.743654 12143 raft_consensus.cc:348] T 8edb24e97d314dc394ac680bc8112117 P 7189161cfb724373947bc63a513b7656 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a6697f25ce748709f4d411ab4e01081" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33073 } } peers { permanent_uuid: "7189161cfb724373947bc63a513b7656" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43215 } } peers { permanent_uuid: "702a429ede7b4d2a9f6b2789bb78655c" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 33647 } }
I20260618 22:19:32.744266 12141 consensus_queue.cc:260] T 8edb24e97d314dc394ac680bc8112117 P 702a429ede7b4d2a9f6b2789bb78655c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a6697f25ce748709f4d411ab4e01081" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33073 } } peers { permanent_uuid: "7189161cfb724373947bc63a513b7656" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43215 } } peers { permanent_uuid: "702a429ede7b4d2a9f6b2789bb78655c" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 33647 } }
I20260618 22:19:32.745282 12141 ts_tablet_manager.cc:1428] T 8edb24e97d314dc394ac680bc8112117 P 702a429ede7b4d2a9f6b2789bb78655c: Time spent starting tablet: real 0.013s user 0.005s sys 0.000s
I20260618 22:19:32.745404 12143 raft_consensus.cc:374] T 8edb24e97d314dc394ac680bc8112117 P 7189161cfb724373947bc63a513b7656 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:19:32.745496 12143 raft_consensus.cc:729] T 8edb24e97d314dc394ac680bc8112117 P 7189161cfb724373947bc63a513b7656 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7189161cfb724373947bc63a513b7656, State: Initialized, Role: FOLLOWER
I20260618 22:19:32.754451 12143 consensus_queue.cc:260] T 8edb24e97d314dc394ac680bc8112117 P 7189161cfb724373947bc63a513b7656 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a6697f25ce748709f4d411ab4e01081" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33073 } } peers { permanent_uuid: "7189161cfb724373947bc63a513b7656" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43215 } } peers { permanent_uuid: "702a429ede7b4d2a9f6b2789bb78655c" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 33647 } }
I20260618 22:19:32.755019 11989 heartbeater.cc:503] Master 127.0.98.126:33901 was elected leader, sending a full tablet report...
I20260618 22:19:32.758534 12143 ts_tablet_manager.cc:1428] T 8edb24e97d314dc394ac680bc8112117 P 7189161cfb724373947bc63a513b7656: Time spent starting tablet: real 0.019s user 0.005s sys 0.001s
I20260618 22:19:32.758886 12123 heartbeater.cc:503] Master 127.0.98.126:33901 was elected leader, sending a full tablet report...
I20260618 22:19:32.786665 12149 raft_consensus.cc:482] T 8edb24e97d314dc394ac680bc8112117 P 7189161cfb724373947bc63a513b7656 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:19:32.786901 12149 raft_consensus.cc:504] T 8edb24e97d314dc394ac680bc8112117 P 7189161cfb724373947bc63a513b7656 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a6697f25ce748709f4d411ab4e01081" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33073 } } peers { permanent_uuid: "7189161cfb724373947bc63a513b7656" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43215 } } peers { permanent_uuid: "702a429ede7b4d2a9f6b2789bb78655c" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 33647 } }
I20260618 22:19:32.788200 12149 leader_election.cc:288] T 8edb24e97d314dc394ac680bc8112117 P 7189161cfb724373947bc63a513b7656 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3a6697f25ce748709f4d411ab4e01081 (127.0.98.65:33073), 702a429ede7b4d2a9f6b2789bb78655c (127.0.98.66:33647)
W20260618 22:19:32.788981 12124 tablet.cc:2369] T 8edb24e97d314dc394ac680bc8112117 P 7189161cfb724373947bc63a513b7656: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:19:32.796337 12125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:19:32.788637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:33073 (local address 127.0.98.67:34865)
0618 22:19:32.790525 (+ 1888us) negotiation.cc:107] Waiting for socket to connect
0618 22:19:32.790541 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 22:19:32.790665 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:19:32.791678 (+ 1013us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:19:32.791681 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:19:32.791701 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:19:32.791825 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:19:32.791832 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:19:32.792915 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:19:32.792919 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:19:32.794335 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:19:32.794347 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:19:32.795876 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:19:32.795880 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:19:32.795985 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:19:32.796012 (+ 27us) client_negotiation.cc:769] Sending connection context
0618 22:19:32.796111 (+ 99us) client_negotiation.cc:240] Negotiation successful
0618 22:19:32.796190 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1793}
I20260618 22:19:32.797686 12140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:19:32.790899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:34865 (local address 127.0.98.65:33073)
0618 22:19:32.791050 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0618 22:19:32.791056 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:19:32.791076 (+ 20us) server_negotiation.cc:407] Connection header received
0618 22:19:32.791150 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:19:32.791154 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:19:32.791209 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:19:32.791314 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:19:32.791977 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:19:32.792753 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:19:32.794501 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:19:32.795620 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:19:32.795738 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:19:32.795811 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0618 22:19:32.797465 (+ 1654us) server_negotiation.cc:299] Negotiation successful
0618 22:19:32.797558 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260618 22:19:32.798341 11794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8edb24e97d314dc394ac680bc8112117" candidate_uuid: "7189161cfb724373947bc63a513b7656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a6697f25ce748709f4d411ab4e01081" is_pre_election: true
I20260618 22:19:32.798708 11794 raft_consensus.cc:2454] T 8edb24e97d314dc394ac680bc8112117 P 3a6697f25ce748709f4d411ab4e01081 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7189161cfb724373947bc63a513b7656 in term 0.
I20260618 22:19:32.805109 12011 leader_election.cc:302] T 8edb24e97d314dc394ac680bc8112117 P 7189161cfb724373947bc63a513b7656 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a6697f25ce748709f4d411ab4e01081, 7189161cfb724373947bc63a513b7656; no voters:
I20260618 22:19:32.805470 12149 raft_consensus.cc:2781] T 8edb24e97d314dc394ac680bc8112117 P 7189161cfb724373947bc63a513b7656 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:19:32.805555 12149 raft_consensus.cc:482] T 8edb24e97d314dc394ac680bc8112117 P 7189161cfb724373947bc63a513b7656 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:19:32.805608 12149 raft_consensus.cc:3037] T 8edb24e97d314dc394ac680bc8112117 P 7189161cfb724373947bc63a513b7656 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:19:32.807060 12149 raft_consensus.cc:504] T 8edb24e97d314dc394ac680bc8112117 P 7189161cfb724373947bc63a513b7656 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a6697f25ce748709f4d411ab4e01081" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33073 } } peers { permanent_uuid: "7189161cfb724373947bc63a513b7656" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43215 } } peers { permanent_uuid: "702a429ede7b4d2a9f6b2789bb78655c" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 33647 } }
I20260618 22:19:32.807512 12149 leader_election.cc:288] T 8edb24e97d314dc394ac680bc8112117 P 7189161cfb724373947bc63a513b7656 [CANDIDATE]: Term 1 election: Requested vote from peers 3a6697f25ce748709f4d411ab4e01081 (127.0.98.65:33073), 702a429ede7b4d2a9f6b2789bb78655c (127.0.98.66:33647)
I20260618 22:19:32.808251 11794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8edb24e97d314dc394ac680bc8112117" candidate_uuid: "7189161cfb724373947bc63a513b7656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3a6697f25ce748709f4d411ab4e01081"
I20260618 22:19:32.808393 11794 raft_consensus.cc:3037] T 8edb24e97d314dc394ac680bc8112117 P 3a6697f25ce748709f4d411ab4e01081 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:19:32.809744 11794 raft_consensus.cc:2454] T 8edb24e97d314dc394ac680bc8112117 P 3a6697f25ce748709f4d411ab4e01081 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7189161cfb724373947bc63a513b7656 in term 1.
I20260618 22:19:32.810213 12011 leader_election.cc:302] T 8edb24e97d314dc394ac680bc8112117 P 7189161cfb724373947bc63a513b7656 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3a6697f25ce748709f4d411ab4e01081, 7189161cfb724373947bc63a513b7656; no voters:
I20260618 22:19:32.810457 12149 raft_consensus.cc:2781] T 8edb24e97d314dc394ac680bc8112117 P 7189161cfb724373947bc63a513b7656 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:19:32.810664 12149 raft_consensus.cc:686] T 8edb24e97d314dc394ac680bc8112117 P 7189161cfb724373947bc63a513b7656 [term 1 LEADER]: Becoming Leader. State: Replica: 7189161cfb724373947bc63a513b7656, State: Running, Role: LEADER
I20260618 22:19:32.811076 12149 consensus_queue.cc:237] T 8edb24e97d314dc394ac680bc8112117 P 7189161cfb724373947bc63a513b7656 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a6697f25ce748709f4d411ab4e01081" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33073 } } peers { permanent_uuid: "7189161cfb724373947bc63a513b7656" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43215 } } peers { permanent_uuid: "702a429ede7b4d2a9f6b2789bb78655c" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 33647 } }
I20260618 22:19:32.819799 12125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:19:32.791292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:33647 (local address 127.0.98.67:44169)
0618 22:19:32.796511 (+ 5219us) negotiation.cc:107] Waiting for socket to connect
0618 22:19:32.796528 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0618 22:19:32.796615 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:19:32.811777 (+ 15162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:19:32.811781 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:19:32.811808 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:19:32.811915 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:19:32.811966 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:19:32.816974 (+ 5008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:19:32.816978 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:19:32.818213 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:19:32.818227 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:19:32.819329 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:19:32.819333 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:19:32.819433 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:19:32.819458 (+ 25us) client_negotiation.cc:769] Sending connection context
0618 22:19:32.819542 (+ 84us) client_negotiation.cc:240] Negotiation successful
0618 22:19:32.819622 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5160,"thread_start_us":113,"threads_started":1}
I20260618 22:19:32.820410 12139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:19:32.791578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:44169 (local address 127.0.98.66:33647)
0618 22:19:32.791813 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0618 22:19:32.791818 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:19:32.796815 (+ 4997us) server_negotiation.cc:407] Connection header received
0618 22:19:32.796904 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:19:32.796908 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:19:32.796969 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:19:32.797052 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:19:32.812152 (+ 15100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:19:32.812971 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:19:32.818387 (+ 5416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:19:32.819010 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:19:32.819122 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:19:32.819200 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0618 22:19:32.820188 (+ 988us) server_negotiation.cc:299] Negotiation successful
0618 22:19:32.820279 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146}
I20260618 22:19:32.822542 11638 catalog_manager.cc:5495] T 8edb24e97d314dc394ac680bc8112117 P 7189161cfb724373947bc63a513b7656 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7189161cfb724373947bc63a513b7656 (127.0.98.67). New cstate: current_term: 1 leader_uuid: "7189161cfb724373947bc63a513b7656" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3a6697f25ce748709f4d411ab4e01081" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33073 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7189161cfb724373947bc63a513b7656" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "702a429ede7b4d2a9f6b2789bb78655c" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 33647 } health_report { overall_health: UNKNOWN } } }
I20260618 22:19:32.825724 11943 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8edb24e97d314dc394ac680bc8112117" candidate_uuid: "7189161cfb724373947bc63a513b7656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "702a429ede7b4d2a9f6b2789bb78655c" is_pre_election: true
I20260618 22:19:32.826061 11943 raft_consensus.cc:2454] T 8edb24e97d314dc394ac680bc8112117 P 702a429ede7b4d2a9f6b2789bb78655c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7189161cfb724373947bc63a513b7656 in term 0.
I20260618 22:19:32.826427 11942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8edb24e97d314dc394ac680bc8112117" candidate_uuid: "7189161cfb724373947bc63a513b7656" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "702a429ede7b4d2a9f6b2789bb78655c"
I20260618 22:19:32.826548 11942 raft_consensus.cc:3037] T 8edb24e97d314dc394ac680bc8112117 P 702a429ede7b4d2a9f6b2789bb78655c [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:19:32.827893 11942 raft_consensus.cc:2454] T 8edb24e97d314dc394ac680bc8112117 P 702a429ede7b4d2a9f6b2789bb78655c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7189161cfb724373947bc63a513b7656 in term 1.
I20260618 22:19:32.872403 12137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:19:32.867186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49772 (local address 127.0.98.67:43215)
0618 22:19:32.867463 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0618 22:19:32.867468 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:19:32.867485 (+ 17us) server_negotiation.cc:407] Connection header received
0618 22:19:32.867575 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:19:32.867578 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:19:32.867656 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:19:32.867778 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:19:32.868237 (+ 459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:19:32.869031 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:19:32.870276 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:19:32.870635 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:19:32.870749 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:19:32.871151 (+ 402us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:19:32.871693 (+ 542us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:19:32.871808 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0618 22:19:32.872192 (+ 384us) server_negotiation.cc:299] Negotiation successful
0618 22:19:32.872248 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201}
I20260618 22:19:32.884722 11942 raft_consensus.cc:1261] T 8edb24e97d314dc394ac680bc8112117 P 702a429ede7b4d2a9f6b2789bb78655c [term 1 FOLLOWER]: Refusing update from remote peer 7189161cfb724373947bc63a513b7656: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:19:32.885702 12149 consensus_queue.cc:1035] T 8edb24e97d314dc394ac680bc8112117 P 7189161cfb724373947bc63a513b7656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "702a429ede7b4d2a9f6b2789bb78655c" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 33647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:19:32.893149 11794 raft_consensus.cc:1261] T 8edb24e97d314dc394ac680bc8112117 P 3a6697f25ce748709f4d411ab4e01081 [term 1 FOLLOWER]: Refusing update from remote peer 7189161cfb724373947bc63a513b7656: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:19:32.894038 12149 consensus_queue.cc:1035] T 8edb24e97d314dc394ac680bc8112117 P 7189161cfb724373947bc63a513b7656 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a6697f25ce748709f4d411ab4e01081" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33073 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260618 22:19:32.903003 11852 tablet.cc:2369] T 8edb24e97d314dc394ac680bc8112117 P 3a6697f25ce748709f4d411ab4e01081: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:19:32.914482 12154 mvcc.cc:205] Tried to move back new op lower bound from 7298339524126482432 to 7298339523853635584. Current Snapshot: MvccSnapshot[applied={T|T < 7298339524126482432}]
I20260618 22:19:32.917971 12157 mvcc.cc:205] Tried to move back new op lower bound from 7298339524126482432 to 7298339523853635584. Current Snapshot: MvccSnapshot[applied={T|T < 7298339524126482432}]
I20260618 22:19:32.924014 11638 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:19:32.924185 11638 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39880:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:19:32.927102 11638 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3db860900e304417be41cb4118793d8a: Sending DeleteTablet for 3 replicas of tablet 8edb24e97d314dc394ac680bc8112117
I20260618 22:19:32.928138 12057 tablet_service.cc:1514] Processing DeleteTablet for tablet 8edb24e97d314dc394ac680bc8112117 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:19:32 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:49770
I20260618 22:19:32.928609 11915 tablet_service.cc:1514] Processing DeleteTablet for tablet 8edb24e97d314dc394ac680bc8112117 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:19:32 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:57776
I20260618 22:19:32.928864 11770 tablet_service.cc:1514] Processing DeleteTablet for tablet 8edb24e97d314dc394ac680bc8112117 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:19:32 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:48132
I20260618 22:19:32.929970 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 11720
I20260618 22:19:32.932247 12165 tablet_replica.cc:330] T 8edb24e97d314dc394ac680bc8112117 P 7189161cfb724373947bc63a513b7656: stopping tablet replica
I20260618 22:19:32.932535 12165 raft_consensus.cc:2229] T 8edb24e97d314dc394ac680bc8112117 P 7189161cfb724373947bc63a513b7656 [term 1 LEADER]: Raft consensus shutting down.
I20260618 22:19:32.932904 12165 raft_consensus.cc:2258] T 8edb24e97d314dc394ac680bc8112117 P 7189161cfb724373947bc63a513b7656 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:19:32.933219 12166 tablet_replica.cc:330] T 8edb24e97d314dc394ac680bc8112117 P 702a429ede7b4d2a9f6b2789bb78655c: stopping tablet replica
I20260618 22:19:32.933780 12166 raft_consensus.cc:2229] T 8edb24e97d314dc394ac680bc8112117 P 702a429ede7b4d2a9f6b2789bb78655c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:19:32.934962 12166 raft_consensus.cc:2258] T 8edb24e97d314dc394ac680bc8112117 P 702a429ede7b4d2a9f6b2789bb78655c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:19:32.938258 12166 ts_tablet_manager.cc:1905] T 8edb24e97d314dc394ac680bc8112117 P 702a429ede7b4d2a9f6b2789bb78655c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:19:32.941186 12165 ts_tablet_manager.cc:1905] T 8edb24e97d314dc394ac680bc8112117 P 7189161cfb724373947bc63a513b7656: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:19:32.945511 12165 ts_tablet_manager.cc:1918] T 8edb24e97d314dc394ac680bc8112117 P 7189161cfb724373947bc63a513b7656: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:19:32.949069 12166 ts_tablet_manager.cc:1918] T 8edb24e97d314dc394ac680bc8112117 P 702a429ede7b4d2a9f6b2789bb78655c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260618 22:19:32.951521 11624 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.65:33073 (error 108)
I20260618 22:19:32.952366 12166 log.cc:1196] T 8edb24e97d314dc394ac680bc8112117 P 702a429ede7b4d2a9f6b2789bb78655c: Deleting WAL directory at /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/ts-1/wal/wals/8edb24e97d314dc394ac680bc8112117
I20260618 22:19:32.953092 12165 log.cc:1196] T 8edb24e97d314dc394ac680bc8112117 P 7189161cfb724373947bc63a513b7656: Deleting WAL directory at /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781821065380026-393-0/minicluster-data/ts-2/wal/wals/8edb24e97d314dc394ac680bc8112117
I20260618 22:19:32.954357 12166 ts_tablet_manager.cc:1939] T 8edb24e97d314dc394ac680bc8112117 P 702a429ede7b4d2a9f6b2789bb78655c: Deleting consensus metadata
I20260618 22:19:32.955175 12165 ts_tablet_manager.cc:1939] T 8edb24e97d314dc394ac680bc8112117 P 7189161cfb724373947bc63a513b7656: Deleting consensus metadata
I20260618 22:19:32.956638 11625 catalog_manager.cc:4841] TS 702a429ede7b4d2a9f6b2789bb78655c (127.0.98.66:33647): tablet 8edb24e97d314dc394ac680bc8112117 (table test-table [id=a649271a8ec5421d8105396916ae7398]) successfully deleted
I20260618 22:19:32.957520 11626 catalog_manager.cc:4841] TS 7189161cfb724373947bc63a513b7656 (127.0.98.67:43215): tablet 8edb24e97d314dc394ac680bc8112117 (table test-table [id=a649271a8ec5421d8105396916ae7398]) successfully deleted
I20260618 22:19:32.958185 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 11859
I20260618 22:19:32.960687 12138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:19:32.960351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:33073 (local address 127.0.0.1:48140)
0618 22:19:32.960522 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0618 22:19:32.960589 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.65:33073: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
W20260618 22:19:32.960925 11624 catalog_manager.cc:4584] TS 3a6697f25ce748709f4d411ab4e01081 (127.0.98.65:33073): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8edb24e97d314dc394ac680bc8112117: Network error: Client connection negotiation failed: client connection to 127.0.98.65:33073: connect: Connection refused (error 111)
I20260618 22:19:32.973161 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 11994
I20260618 22:19:32.983474 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 11607
I20260618 22:19:32.990597 393 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260618 22:19:33.325053 393 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260618 22:19:33.325171 393 mini_ranger.cc:66] Stopping Ranger...
I20260618 22:19:33.669755 393 mini_ranger.cc:68] Stopped Ranger
2026-06-18 22:19:33.670 UTC [11193] LOG: received smart shutdown request
2026-06-18 22:19:33.673 UTC [11193] LOG: background worker "logical replication launcher" (PID 11202) exited with exit code 1
2026-06-18 22:19:33.673 UTC [11197] LOG: shutting down
2026-06-18 22:19:33.829 UTC [11193] LOG: database system is shut down
2026-06-18T22:19:33Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (45342 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:19:33 dist-test-slave-g6r4 krb5kdc[12178](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:19:33 dist-test-slave-g6r4 krb5kdc[12178](info): set up 2 sockets
Jun 18 22:19:33 dist-test-slave-g6r4 krb5kdc[12178](info): commencing operation
krb5kdc: starting...
W20260618 22:19:34.974207 393 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/krb5kdc/test-user.keytab.
Jun 18 22:19:35 dist-test-slave-g6r4 krb5kdc[12178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:19:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:19:35Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/krb5kdc/rangeradmin_127.0.98.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/krb5kdc/rangeradmin_127.0.98.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/krb5kdc/rangerlookup_127.0.98.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/krb5kdc/rangerlookup_127.0.98.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/krb5kdc/HTTP_127.0.98.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/krb5kdc/HTTP_127.0.98.84@KRBTEST.COM.keytab.
I20260618 22:19:35.132400 393 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskIMAyAx/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/postgres -l logfile start
2026-06-18 22:19:37.559 UTC [12208] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-18 22:19:37.559 UTC [12208] LOG: listening on IPv4 address "127.0.98.84", port 47133
2026-06-18 22:19:37.563 UTC [12208] LOG: listening on Unix socket "/tmp/.s.PGSQL.47133"
2026-06-18 22:19:37.583 UTC [12211] LOG: database system was shut down at 2026-06-18 22:19:36 UTC
2026-06-18 22:19:37.587 UTC [12208] LOG: database system is ready to accept connections
I20260618 22:19:38.573809 393 mini_postgres.cc:96] Postgres bound to 47133
2026-06-18 22:19:38.579 UTC [12219] FATAL: database "slave" does not exist
127.0.98.84:47133 - accepting connections
I20260618 22:19:38.580345 393 mini_ranger.cc:161] Starting Ranger...
I20260618 22:19:38.594511 393 mini_ranger.cc:84] Created miniranger Postgres user
I20260618 22:19:39.062534 393 mini_ranger.cc:87] Created ranger Postgres database
I20260618 22:19:39.062600 393 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ranger-admin
2026-06-18 22:19:39,334 [I] DB FLAVOR :POSTGRES
2026-06-18 22:19:39,334 [I] --------- Verifying Ranger DB connection ---------
2026-06-18 22:19:39,334 [I] Checking connection..
2026-06-18 22:19:39,335 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIMAyAx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIMAyAx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:47133/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-18 22:19:39,586 [I] Checking connection passed.
2026-06-18 22:19:39,586 [I] --------- Verifying version history table ---------
2026-06-18 22:19:39,586 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIMAyAx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIMAyAx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:47133/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-18 22:19:39,808 [I] Table x_db_version_h does not exist in database ranger
2026-06-18 22:19:39,809 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIMAyAx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIMAyAx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:47133/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-18 22:19:40,044 [I] Table x_db_version_h does not exist in database ranger
2026-06-18 22:19:40,044 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-18 22:19:40,044 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIMAyAx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIMAyAx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:47133/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskIMAyAx/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-18 22:19:40.289 UTC [12283] WARNING: there is no transaction in progress
2026-06-18 22:19:40,295 [I] create_dbversion_catalog.sql file imported successfully
2026-06-18 22:19:40,296 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIMAyAx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIMAyAx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:47133/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-18 22:19:40,524 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-18 22:19:40,524 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-18 22:19:40,524 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIMAyAx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIMAyAx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:47133/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-18 22:19:40,751 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIMAyAx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIMAyAx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:47133/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-18 22:19:40,981 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIMAyAx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIMAyAx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:47133/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-g6r4.c.gcp-upstream.internal','N') ;"
2026-06-18 22:19:41,201 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIMAyAx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIMAyAx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:47133/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-18 22:19:41,427 [I] Table x_portal_user does not exist in database ranger
2026-06-18 22:19:41,428 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIMAyAx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIMAyAx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:47133/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-18 22:19:41,667 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-18 22:19:41,668 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIMAyAx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIMAyAx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:47133/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-18 22:19:41,922 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIMAyAx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIMAyAx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:47133/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-18 22:19:42,170 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-18 22:19:42,170 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIMAyAx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIMAyAx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:47133/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskIMAyAx/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-18 22:19:43.112 UTC [12391] WARNING: there is no transaction in progress
2026-06-18 22:19:43.128 UTC [12391] WARNING: there is no transaction in progress
2026-06-18 22:19:43.143 UTC [12391] WARNING: there is no transaction in progress
2026-06-18 22:19:43.158 UTC [12391] WARNING: there is no transaction in progress
2026-06-18 22:19:43.174 UTC [12391] WARNING: there is no transaction in progress
2026-06-18 22:19:43.192 UTC [12391] WARNING: there is no transaction in progress
2026-06-18 22:19:43.243 UTC [12391] WARNING: there is no transaction in progress
2026-06-18 22:19:43.249 UTC [12391] WARNING: there is no transaction in progress
2026-06-18 22:19:43.264 UTC [12391] WARNING: there is no transaction in progress
2026-06-18 22:19:43.274 UTC [12391] WARNING: there is no transaction in progress
2026-06-18 22:19:43.284 UTC [12391] WARNING: there is no transaction in progress
2026-06-18 22:19:43.297 UTC [12391] WARNING: there is no transaction in progress
2026-06-18 22:19:43.306 UTC [12391] WARNING: there is no transaction in progress
2026-06-18 22:19:43.314 UTC [12391] WARNING: there is no transaction in progress
2026-06-18 22:19:43.321 UTC [12391] WARNING: there is no transaction in progress
2026-06-18 22:19:43.328 UTC [12391] WARNING: there is no transaction in progress
2026-06-18 22:19:43,941 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-18 22:19:43,941 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIMAyAx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIMAyAx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:47133/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-18 22:19:44,168 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-18 22:19:44,168 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIMAyAx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIMAyAx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:47133/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-18 22:19:44,403 [I] Table x_portal_user already exists in database 'ranger'
2026-06-18 22:19:44,403 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIMAyAx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIMAyAx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:47133/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-18 22:19:44,639 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-18 22:19:44,639 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIMAyAx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIMAyAx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:47133/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-18 22:19:44,876 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIMAyAx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIMAyAx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:47133/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-18 22:19:45,127 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIMAyAx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIMAyAx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:47133/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-g6r4.c.gcp-upstream.internal';"
2026-06-18 22:19:45,357 [I] CORE_DB_SCHEMA import status has been updated
2026-06-18 22:19:45,357 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIMAyAx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIMAyAx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:47133/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-18 22:19:45,579 [I] DB_PATCHES have already been applied
I20260618 22:19:45.584618 393 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ranger-admin:/tmp/dist-test-taskIMAyAx/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskIMAyAx/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskIMAyAx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIMAyAx/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260618 22:19:45.584677 393 mini_ranger.cc:193] Using host: 127.0.98.84
I20260618 22:19:45.587909 393 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.84:56133
Jun 18, 2026 10:19:45 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 18, 2026 10:19:45 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 18, 2026 10:19:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ranger-admin/ews/webapp
Jun 18, 2026 10:19:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 18, 2026 10:19:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 18, 2026 10:19:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ranger-admin/ews/webapp].
Jun 18, 2026 10:19:46 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260618 22:19:46.736976 393 mini_ranger.cc:160] Time spent starting Ranger: real 8.157s user 0.002s sys 0.007s
Jun 18, 2026 10:20:07 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 18, 2026 10:20:07 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 18, 2026 10:20:07 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 18, 2026 10:20:07 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 18, 2026 10:20:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 18, 2026 10:20:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 18, 2026 10:20:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 18, 2026 10:20:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 18, 2026 10:20:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 18, 2026 10:20:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 18, 2026 10:20:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 18, 2026 10:20:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 18, 2026 10:20:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 18, 2026 10:20:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 18, 2026 10:20:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 18, 2026 10:20:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 18, 2026 10:20:07 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 18, 2026 10:20:07 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260618 22:20:08.451022 393 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/krb5kdc/rangerkms_127.0.98.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/krb5kdc/rangerkms_127.0.98.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/krb5kdc/HTTP_127.0.98.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/krb5kdc/HTTP_127.0.98.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 18 22:20:08 dist-test-slave-g6r4 krb5kdc[12178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821208, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260618 22:20:08.529718 393 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260618 22:20:08.538415 393 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260618 22:20:09.332512 393 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260618 22:20:09.332623 393 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ranger-kms
I20260618 22:20:09.332645 393 mini_ranger_kms.cc:227] Using postgres at 127.0.98.84:47133
2026-06-18 22:20:09,497 [I] DB FLAVOR :POSTGRES
2026-06-18 22:20:09,498 [I] --------- Verifying Ranger DB connection ---------
2026-06-18 22:20:09,498 [I] Checking connection
2026-06-18 22:20:09,498 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIMAyAx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:47133/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-18 22:20:09,722 [I] connection success
2026-06-18 22:20:09,723 [I] --------- Verifying Ranger DB tables ---------
2026-06-18 22:20:09,723 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-18 22:20:09,723 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIMAyAx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:47133/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-18 22:20:09,953 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-18 22:20:09,954 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-18 22:20:09,954 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-18 22:20:09,954 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskIMAyAx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.84:47133/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-18 22:20:10,205 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260618 22:20:10.643222 393 mini_ranger_kms.cc:310] Created kms service
I20260618 22:20:10.833420 393 mini_ranger_kms.cc:326] Created kudu user
I20260618 22:20:10.875720 393 mini_ranger_kms.cc:343] Created rangerkms user
Jun 18 22:20:10 dist-test-slave-g6r4 krb5kdc[12178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821208, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.84@KRBTEST.COM
I20260618 22:20:11.247068 393 mini_ranger_kms.cc:384] Added ranger policy
I20260618 22:20:11.247303 393 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ranger-kms:/tmp/dist-test-taskIMAyAx/build/debug/bin/postgresql.jar:/tmp/dist-test-taskIMAyAx/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskIMAyAx/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260618 22:20:11.247329 393 mini_ranger_kms.cc:240] Using host: 127.0.98.84
I20260618 22:20:11.251200 393 mini_ranger_kms.cc:276] Ranger KMS PID: 12571
I20260618 22:20:11.251291 393 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.84:55441
Jun 18, 2026 10:20:12 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 18, 2026 10:20:12 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 18, 2026 10:20:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 18, 2026 10:20:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 18, 2026 10:20:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ranger-kms/ews/webapp].
Jun 18, 2026 10:20:12 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 18 22:20:13 dist-test-slave-g6r4 krb5kdc[12178](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781821213, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:20:13.490236 393 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.961s user 0.005s sys 0.144s
I20260618 22:20:13.490355 393 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260618 22:20:13.490419 393 mini_ranger_kms.cc:401] 127.0.98.84:55441/kms/v1/keys
Jun 18 22:20:13 dist-test-slave-g6r4 krb5kdc[12178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821213, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.84@KRBTEST.COM for HTTP/127.0.98.84@KRBTEST.COM
Jun 18 22:20:14 dist-test-slave-g6r4 krb5kdc[12178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821213, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.84@KRBTEST.COM for HTTP/127.0.98.84@KRBTEST.COM
Jun 18 22:20:16 dist-test-slave-g6r4 krb5kdc[12178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260618 22:20:16.252310 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:34303
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:35427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:34303
--ranger_config_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.84:55441/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:20:16.384068 12622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:20:16.384296 12622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:20:16.384334 12622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:20:16.388307 12622 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:20:16.388356 12622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:20:16.388383 12622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:20:16.388428 12622 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:20:16.388456 12622 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:20:16.393698 12622 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:35427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.84:55441/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:34303
--ranger_config_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:34303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12622
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:20:16.394697 12622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:20:16.395653 12622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:20:16.403987 12627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:20:16.404000 12630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:20:16.404084 12622 server_base.cc:1034] running on GCE node
W20260618 22:20:16.404115 12628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:20:16.404749 12622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:20:16.405753 12622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:20:16.406932 12622 hybrid_clock.cc:648] HybridClock initialized: now 1781821216406927 us; error 60 us; skew 500 ppm
Jun 18 22:20:16 dist-test-slave-g6r4 krb5kdc[12178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821216, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:20:16.415740 12622 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260618 22:20:16.416602 12622 webserver.cc:460] Webserver started at http://127.0.98.126:34747/ using document root <none> and password file <none>
I20260618 22:20:16.417081 12622 fs_manager.cc:359] Metadata directory not provided
I20260618 22:20:16.417131 12622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:20:16.417289 12622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 18 22:20:16 dist-test-slave-g6r4 krb5kdc[12178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821216, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for HTTP/127.0.98.84@KRBTEST.COM
I20260618 22:20:16.471207 12622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/master-0/data/instance:
uuid: "3096d8ecc01b437a8f0beabf8fb021f7"
format_stamp: "Formatted at 2026-06-18 22:20:16 on dist-test-slave-g6r4"
server_key: "d3df98ecd4b980cc7935d149ba34685e"
server_key_iv: "6029f6cc886ce2b9bd54b8114accc417"
server_key_version: "kuduclusterkey@0"
I20260618 22:20:16.471833 12622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/master-0/wal/instance:
uuid: "3096d8ecc01b437a8f0beabf8fb021f7"
format_stamp: "Formatted at 2026-06-18 22:20:16 on dist-test-slave-g6r4"
server_key: "d3df98ecd4b980cc7935d149ba34685e"
server_key_iv: "6029f6cc886ce2b9bd54b8114accc417"
server_key_version: "kuduclusterkey@0"
I20260618 22:20:16.475822 12622 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260618 22:20:16.492167 12637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:20:16.493530 12622 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260618 22:20:16.493680 12622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/master-0/wal
uuid: "3096d8ecc01b437a8f0beabf8fb021f7"
format_stamp: "Formatted at 2026-06-18 22:20:16 on dist-test-slave-g6r4"
server_key: "d3df98ecd4b980cc7935d149ba34685e"
server_key_iv: "6029f6cc886ce2b9bd54b8114accc417"
server_key_version: "kuduclusterkey@0"
I20260618 22:20:16.493818 12622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:20:16.517376 12622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:20:16.520073 12622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:20:16.520269 12622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:20:16.527683 12622 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:34303
I20260618 22:20:16.527709 12689 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:34303 every 8 connection(s)
I20260618 22:20:16.528569 12622 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
I20260618 22:20:16.529743 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 12622
I20260618 22:20:16.529870 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/master-0/wal/instance
I20260618 22:20:16.530184 393 external_mini_cluster.cc:1428] Setting key f9f5b2c6fe93aae6531ffb63901e4274
I20260618 22:20:16.531735 12690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:20:16.537195 12690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3096d8ecc01b437a8f0beabf8fb021f7: Bootstrap starting.
Jun 18 22:20:16 dist-test-slave-g6r4 krb5kdc[12178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:20:16.539399 12690 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3096d8ecc01b437a8f0beabf8fb021f7: Neither blocks nor log segments found. Creating new log.
I20260618 22:20:16.540501 12690 log.cc:824] T 00000000000000000000000000000000 P 3096d8ecc01b437a8f0beabf8fb021f7: Log is configured to *not* fsync() on all Append() calls
I20260618 22:20:16.542654 12690 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3096d8ecc01b437a8f0beabf8fb021f7: No bootstrap required, opened a new log
I20260618 22:20:16.543668 12693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:16.531508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48930 (local address 127.0.98.126:34303)
0618 22:20:16.531951 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0618 22:20:16.531965 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0618 22:20:16.531999 (+ 34us) server_negotiation.cc:407] Connection header received
0618 22:20:16.532643 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:20:16.532659 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:20:16.532962 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:20:16.533282 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:20:16.534151 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:16.535131 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:16.535967 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:16.536418 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:16.536590 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:16.539003 (+ 2413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:20:16.539008 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:20:16.539020 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:20:16.539044 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:20:16.540640 (+ 1596us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:16.541167 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:16.541172 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:16.541177 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:16.541350 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:16.541635 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:16.541639 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:16.541640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:16.542110 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:20:16.542314 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0618 22:20:16.542530 (+ 216us) server_negotiation.cc:299] Negotiation successful
0618 22:20:16.542816 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":134,"threads_started":1}
I20260618 22:20:16.546115 12690 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3096d8ecc01b437a8f0beabf8fb021f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3096d8ecc01b437a8f0beabf8fb021f7" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34303 } }
I20260618 22:20:16.546299 12690 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3096d8ecc01b437a8f0beabf8fb021f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:20:16.546332 12690 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3096d8ecc01b437a8f0beabf8fb021f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3096d8ecc01b437a8f0beabf8fb021f7, State: Initialized, Role: FOLLOWER
I20260618 22:20:16.546685 12690 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3096d8ecc01b437a8f0beabf8fb021f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3096d8ecc01b437a8f0beabf8fb021f7" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34303 } }
I20260618 22:20:16.546783 12690 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3096d8ecc01b437a8f0beabf8fb021f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:20:16.546825 12690 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3096d8ecc01b437a8f0beabf8fb021f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:20:16.546896 12690 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3096d8ecc01b437a8f0beabf8fb021f7 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:20:16.547621 12690 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3096d8ecc01b437a8f0beabf8fb021f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3096d8ecc01b437a8f0beabf8fb021f7" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34303 } }
I20260618 22:20:16.547896 12690 leader_election.cc:302] T 00000000000000000000000000000000 P 3096d8ecc01b437a8f0beabf8fb021f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3096d8ecc01b437a8f0beabf8fb021f7; no voters:
I20260618 22:20:16.548121 12690 leader_election.cc:288] T 00000000000000000000000000000000 P 3096d8ecc01b437a8f0beabf8fb021f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:20:16.548208 12695 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3096d8ecc01b437a8f0beabf8fb021f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:20:16.548449 12695 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3096d8ecc01b437a8f0beabf8fb021f7 [term 1 LEADER]: Becoming Leader. State: Replica: 3096d8ecc01b437a8f0beabf8fb021f7, State: Running, Role: LEADER
I20260618 22:20:16.548797 12695 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3096d8ecc01b437a8f0beabf8fb021f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3096d8ecc01b437a8f0beabf8fb021f7" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34303 } }
I20260618 22:20:16.549259 12690 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3096d8ecc01b437a8f0beabf8fb021f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:20:16.550432 12696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3096d8ecc01b437a8f0beabf8fb021f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3096d8ecc01b437a8f0beabf8fb021f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3096d8ecc01b437a8f0beabf8fb021f7" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34303 } } }
I20260618 22:20:16.550547 12696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3096d8ecc01b437a8f0beabf8fb021f7 [sys.catalog]: This master's current role is: LEADER
I20260618 22:20:16.550860 12697 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3096d8ecc01b437a8f0beabf8fb021f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3096d8ecc01b437a8f0beabf8fb021f7. Latest consensus state: current_term: 1 leader_uuid: "3096d8ecc01b437a8f0beabf8fb021f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3096d8ecc01b437a8f0beabf8fb021f7" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34303 } } }
I20260618 22:20:16.550964 12697 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3096d8ecc01b437a8f0beabf8fb021f7 [sys.catalog]: This master's current role is: LEADER
I20260618 22:20:16.551308 12704 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:20:16.551481 12690 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260618 22:20:16.556169 12704 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:20:16.562732 12704 catalog_manager.cc:1347] Generated new cluster ID: 68b8d6d2b17547b4aec69dcf2be39c00
I20260618 22:20:16.562784 12704 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:20:16.575655 12704 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:20:16.576864 12704 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:20:16.590829 12704 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3096d8ecc01b437a8f0beabf8fb021f7: Generated new TSK 0
I20260618 22:20:16.591714 12704 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab.
I20260618 22:20:17.269200 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.84:55441/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:34303
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:35427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:20:17.400069 12735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:20:17.400449 12735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:20:17.400530 12735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:20:17.406091 12735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:20:17.406149 12735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:20:17.406245 12735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260618 22:20:17.412784 12735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:35427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.84:55441/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:34303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:20:17.413874 12735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:20:17.415042 12735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:20:17.424427 12743 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:20:17.424439 12742 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:20:17.424464 12745 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:20:17.425290 12735 server_base.cc:1034] running on GCE node
I20260618 22:20:17.425691 12735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:20:17.426440 12735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:20:17.427611 12735 hybrid_clock.cc:648] HybridClock initialized: now 1781821217427598 us; error 38 us; skew 500 ppm
Jun 18 22:20:17 dist-test-slave-g6r4 krb5kdc[12178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:20:17.430646 12735 init.cc:377] Logged in from keytab as kudu/127.0.98.65@KRBTEST.COM (short username kudu)
I20260618 22:20:17.431675 12735 webserver.cc:460] Webserver started at http://127.0.98.65:43881/ using document root <none> and password file <none>
I20260618 22:20:17.432385 12735 fs_manager.cc:359] Metadata directory not provided
I20260618 22:20:17.432479 12735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:20:17.432785 12735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 18 22:20:17 dist-test-slave-g6r4 krb5kdc[12178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for HTTP/127.0.98.84@KRBTEST.COM
I20260618 22:20:17.456099 12735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-0/data/instance:
uuid: "7d0f3194f19141c697b072ecc2faea28"
format_stamp: "Formatted at 2026-06-18 22:20:17 on dist-test-slave-g6r4"
server_key: "0341ddf91565988f7e0934f7308d6731"
server_key_iv: "66ee6e273a49d646996a063baa9e8bd2"
server_key_version: "kuduclusterkey@0"
I20260618 22:20:17.456872 12735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-0/wal/instance:
uuid: "7d0f3194f19141c697b072ecc2faea28"
format_stamp: "Formatted at 2026-06-18 22:20:17 on dist-test-slave-g6r4"
server_key: "0341ddf91565988f7e0934f7308d6731"
server_key_iv: "66ee6e273a49d646996a063baa9e8bd2"
server_key_version: "kuduclusterkey@0"
I20260618 22:20:17.461608 12735 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260618 22:20:17.486394 12752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:20:17.488054 12735 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.002s sys 0.000s
I20260618 22:20:17.488178 12735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-0/wal
uuid: "7d0f3194f19141c697b072ecc2faea28"
format_stamp: "Formatted at 2026-06-18 22:20:17 on dist-test-slave-g6r4"
server_key: "0341ddf91565988f7e0934f7308d6731"
server_key_iv: "66ee6e273a49d646996a063baa9e8bd2"
server_key_version: "kuduclusterkey@0"
I20260618 22:20:17.488305 12735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:20:17.548183 12735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:20:17.551935 12735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:20:17.552220 12735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:20:17.552914 12735 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:20:17.553913 12735 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:20:17.554016 12735 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:20:17.554128 12735 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:20:17.554198 12735 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:20:17.565399 12735 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:40163
I20260618 22:20:17.566792 12735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
I20260618 22:20:17.567265 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 12735
I20260618 22:20:17.567404 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-0/wal/instance
I20260618 22:20:17.567711 393 external_mini_cluster.cc:1428] Setting key 296bf7d33f4fb2a554231edd1aa74d1b
I20260618 22:20:17.568913 12867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:40163 every 8 connection(s)
Jun 18 22:20:17 dist-test-slave-g6r4 krb5kdc[12178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:20:17.595041 12872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:17.573352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:38831 (local address 127.0.98.126:34303)
0618 22:20:17.575430 (+ 2078us) server_negotiation.cc:206] Beginning negotiation
0618 22:20:17.575439 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0618 22:20:17.575454 (+ 15us) server_negotiation.cc:407] Connection header received
0618 22:20:17.575495 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:20:17.575498 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:20:17.575545 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:20:17.575626 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:20:17.577170 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:17.578154 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:17.579978 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:17.580382 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:17.580468 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:17.585467 (+ 4999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:20:17.585471 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:20:17.585474 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:20:17.585509 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:20:17.587490 (+ 1981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:17.588118 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:17.588122 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:17.588124 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:17.588215 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:17.593290 (+ 5075us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:17.593295 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:17.593300 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:17.593625 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:20:17.593710 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0618 22:20:17.594676 (+ 966us) server_negotiation.cc:299] Negotiation successful
0618 22:20:17.594828 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1996,"thread_start_us":91,"threads_started":1}
I20260618 22:20:17.596645 12871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:17.572124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34303 (local address 127.0.98.65:38831)
0618 22:20:17.573838 (+ 1714us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:17.573928 (+ 90us) client_negotiation.cc:174] Beginning negotiation
0618 22:20:17.574983 (+ 1055us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:20:17.575932 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:20:17.575941 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:20:17.576288 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:20:17.576999 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:17.577019 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:17.578308 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:17.578313 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:17.579821 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:17.579842 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:17.580679 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:17.580683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:17.580906 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:17.581678 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:20:17.581700 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:20:17.585261 (+ 3561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:20:17.587633 (+ 2372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:20:17.587639 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:20:17.587653 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:20:17.587996 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:20:17.588323 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:20:17.588326 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:20:17.588328 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:20:17.588511 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:20:17.593802 (+ 5291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:20:17.593809 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:20:17.594191 (+ 382us) client_negotiation.cc:769] Sending connection context
0618 22:20:17.595107 (+ 916us) client_negotiation.cc:240] Negotiation successful
0618 22:20:17.595562 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1483,"thread_start_us":170,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.66@KRBTEST.COM; defaulting to no policy
I20260618 22:20:17.598315 12868 heartbeater.cc:347] Connected to a master server at 127.0.98.126:34303
I20260618 22:20:17.598570 12868 heartbeater.cc:464] Registering TS with master...
I20260618 22:20:17.599146 12868 heartbeater.cc:511] Master 127.0.98.126:34303 requested a full tablet report, sending...
I20260618 22:20:17.600944 12654 ts_manager.cc:194] Registered new tserver with Master: 7d0f3194f19141c697b072ecc2faea28 (127.0.98.65:40163)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.66@KRBTEST.COM" created.
I20260618 22:20:17.602492 12654 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.65@KRBTEST.COM'} at 127.0.98.65:38831
Jun 18 22:20:17 dist-test-slave-g6r4 krb5kdc[12178](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781821217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab.
I20260618 22:20:17.620983 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.84:55441/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:34303
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:35427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:20:17.788065 12874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:20:17.788489 12874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:20:17.788623 12874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:20:17.794924 12874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:20:17.794998 12874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:20:17.795089 12874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260618 22:20:17.803158 12874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:35427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.84:55441/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:34303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12874
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:20:17.804667 12874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:20:17.806059 12874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:20:17.817323 12881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:20:17.817327 12880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:20:17.817299 12883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:20:17.818241 12874 server_base.cc:1034] running on GCE node
I20260618 22:20:17.818606 12874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:20:17.819299 12874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:20:17.822564 12874 hybrid_clock.cc:648] HybridClock initialized: now 1781821217822532 us; error 46 us; skew 500 ppm
Jun 18 22:20:17 dist-test-slave-g6r4 krb5kdc[12178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:20:17.825506 12874 init.cc:377] Logged in from keytab as kudu/127.0.98.66@KRBTEST.COM (short username kudu)
I20260618 22:20:17.826644 12874 webserver.cc:460] Webserver started at http://127.0.98.66:44021/ using document root <none> and password file <none>
I20260618 22:20:17.827333 12874 fs_manager.cc:359] Metadata directory not provided
I20260618 22:20:17.827428 12874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:20:17.827690 12874 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 18 22:20:17 dist-test-slave-g6r4 krb5kdc[12178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for HTTP/127.0.98.84@KRBTEST.COM
I20260618 22:20:17.843572 12874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-1/data/instance:
uuid: "4e4d9d282a65497e9189190a01a26bab"
format_stamp: "Formatted at 2026-06-18 22:20:17 on dist-test-slave-g6r4"
server_key: "9fe0a17760b6a96515761025c134aa74"
server_key_iv: "f718f66c93924d9f4eb1e8dcc952a43e"
server_key_version: "kuduclusterkey@0"
I20260618 22:20:17.844300 12874 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-1/wal/instance:
uuid: "4e4d9d282a65497e9189190a01a26bab"
format_stamp: "Formatted at 2026-06-18 22:20:17 on dist-test-slave-g6r4"
server_key: "9fe0a17760b6a96515761025c134aa74"
server_key_iv: "f718f66c93924d9f4eb1e8dcc952a43e"
server_key_version: "kuduclusterkey@0"
I20260618 22:20:17.849346 12874 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260618 22:20:17.866160 12890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:20:17.867763 12874 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:20:17.867939 12874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-1/wal
uuid: "4e4d9d282a65497e9189190a01a26bab"
format_stamp: "Formatted at 2026-06-18 22:20:17 on dist-test-slave-g6r4"
server_key: "9fe0a17760b6a96515761025c134aa74"
server_key_iv: "f718f66c93924d9f4eb1e8dcc952a43e"
server_key_version: "kuduclusterkey@0"
I20260618 22:20:17.868111 12874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:20:17.911517 12874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:20:17.920091 12874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:20:17.920346 12874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:20:17.921093 12874 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:20:17.922219 12874 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:20:17.922323 12874 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:20:17.922421 12874 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:20:17.922482 12874 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:20:17.946506 12874 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:34817
I20260618 22:20:17.947695 12874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
I20260618 22:20:17.948803 13003 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:34817 every 8 connection(s)
I20260618 22:20:17.949988 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 12874
I20260618 22:20:17.950114 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-1/wal/instance
I20260618 22:20:17.950446 393 external_mini_cluster.cc:1428] Setting key b5ca8b5d4a9c834f3f5c3a0feb1e805e
Jun 18 22:20:17 dist-test-slave-g6r4 krb5kdc[12178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:20:17.970549 12872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:17.954013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:43241 (local address 127.0.98.126:34303)
0618 22:20:17.954148 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0618 22:20:17.954153 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:20:17.957621 (+ 3468us) server_negotiation.cc:407] Connection header received
0618 22:20:17.958821 (+ 1200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:20:17.958824 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:20:17.958878 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:20:17.958929 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:20:17.960482 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:17.961300 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:17.962993 (+ 1693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:17.963331 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:17.963422 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:17.966489 (+ 3067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:20:17.966493 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:20:17.966496 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:20:17.966522 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:20:17.968310 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:17.968881 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:17.968885 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:17.968887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:17.968970 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:17.969354 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:17.969358 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:17.969359 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:17.969548 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:20:17.969620 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0618 22:20:17.970269 (+ 649us) server_negotiation.cc:299] Negotiation successful
0618 22:20:17.970399 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
WARNING: no policy specified for kudu/127.0.98.67@KRBTEST.COM; defaulting to no policy
I20260618 22:20:17.972195 13006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:17.953333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34303 (local address 127.0.98.66:43241)
0618 22:20:17.957467 (+ 4134us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:17.957526 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0618 22:20:17.958582 (+ 1056us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:20:17.959328 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:20:17.959337 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:20:17.959684 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:20:17.960308 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:17.960326 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:17.961468 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:17.961471 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:17.962860 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:17.962870 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:17.963614 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:17.963617 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:17.963812 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:17.964511 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:20:17.964530 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:20:17.966293 (+ 1763us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:20:17.968456 (+ 2163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:20:17.968461 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:20:17.968475 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:20:17.968772 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:20:17.969068 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:20:17.969070 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:20:17.969072 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:20:17.969236 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:20:17.969679 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:20:17.969684 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:20:17.969987 (+ 303us) client_negotiation.cc:769] Sending connection context
0618 22:20:17.970909 (+ 922us) client_negotiation.cc:240] Negotiation successful
0618 22:20:17.971210 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3888,"thread_start_us":147,"threads_started":1}
I20260618 22:20:17.973624 13004 heartbeater.cc:347] Connected to a master server at 127.0.98.126:34303
I20260618 22:20:17.973855 13004 heartbeater.cc:464] Registering TS with master...
I20260618 22:20:17.974429 13004 heartbeater.cc:511] Master 127.0.98.126:34303 requested a full tablet report, sending...
I20260618 22:20:17.975519 12654 ts_manager.cc:194] Registered new tserver with Master: 4e4d9d282a65497e9189190a01a26bab (127.0.98.66:34817)
I20260618 22:20:17.976305 12654 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.66@KRBTEST.COM'} at 127.0.98.66:43241
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab.
I20260618 22:20:18.011709 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.84:55441/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:34303
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:35427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 18 22:20:18 dist-test-slave-g6r4 krb5kdc[12178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821217, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for HTTP/127.0.98.84@KRBTEST.COM
W20260618 22:20:18.191256 13009 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:20:18.191743 13009 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:20:18.191855 13009 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:20:18.197156 13009 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:20:18.197242 13009 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:20:18.197381 13009 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260618 22:20:18.204818 13009 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:35427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.84:55441/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:34303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13009
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:20:18.206285 13009 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:20:18.207562 13009 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:20:18.220057 13015 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:20:18.220165 13014 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:20:18.220167 13017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:20:18.221220 13009 server_base.cc:1034] running on GCE node
I20260618 22:20:18.221638 13009 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:20:18.222385 13009 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:20:18.229784 13009 hybrid_clock.cc:648] HybridClock initialized: now 1781821218229734 us; error 52 us; skew 500 ppm
Jun 18 22:20:18 dist-test-slave-g6r4 krb5kdc[12178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821218, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:20:18.232779 13009 init.cc:377] Logged in from keytab as kudu/127.0.98.67@KRBTEST.COM (short username kudu)
I20260618 22:20:18.233582 13009 webserver.cc:460] Webserver started at http://127.0.98.67:38969/ using document root <none> and password file <none>
I20260618 22:20:18.234227 13009 fs_manager.cc:359] Metadata directory not provided
I20260618 22:20:18.234326 13009 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:20:18.234539 13009 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 18 22:20:18 dist-test-slave-g6r4 krb5kdc[12178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821218, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for HTTP/127.0.98.84@KRBTEST.COM
I20260618 22:20:18.247498 13009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-2/data/instance:
uuid: "3ba34efe46904a9581b54eea2f6b6cda"
format_stamp: "Formatted at 2026-06-18 22:20:18 on dist-test-slave-g6r4"
server_key: "838a0633ae8c5ba5d36a6d8c58571d1c"
server_key_iv: "ad8a9cf7e2ec5e6daa66bb90867dd193"
server_key_version: "kuduclusterkey@0"
I20260618 22:20:18.248287 13009 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-2/wal/instance:
uuid: "3ba34efe46904a9581b54eea2f6b6cda"
format_stamp: "Formatted at 2026-06-18 22:20:18 on dist-test-slave-g6r4"
server_key: "838a0633ae8c5ba5d36a6d8c58571d1c"
server_key_iv: "ad8a9cf7e2ec5e6daa66bb90867dd193"
server_key_version: "kuduclusterkey@0"
I20260618 22:20:18.252561 13009 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260618 22:20:18.279479 13024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:20:18.281119 13009 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260618 22:20:18.281237 13009 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-2/wal
uuid: "3ba34efe46904a9581b54eea2f6b6cda"
format_stamp: "Formatted at 2026-06-18 22:20:18 on dist-test-slave-g6r4"
server_key: "838a0633ae8c5ba5d36a6d8c58571d1c"
server_key_iv: "ad8a9cf7e2ec5e6daa66bb90867dd193"
server_key_version: "kuduclusterkey@0"
I20260618 22:20:18.281373 13009 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:20:18.301198 13009 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:20:18.304186 13009 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:20:18.304379 13009 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:20:18.304980 13009 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:20:18.305835 13009 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:20:18.305928 13009 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:20:18.306048 13009 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:20:18.306092 13009 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:20:18.316560 13009 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:35107
I20260618 22:20:18.316708 13137 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:35107 every 8 connection(s)
I20260618 22:20:18.317478 13009 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
I20260618 22:20:18.324211 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 13009
I20260618 22:20:18.324344 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-2/wal/instance
I20260618 22:20:18.324620 393 external_mini_cluster.cc:1428] Setting key a9a02c1984a6718ff94047a6727d3736
Jun 18 22:20:18 dist-test-slave-g6r4 krb5kdc[12178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821218, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:20:18.331552 12872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:18.319308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:50959 (local address 127.0.98.126:34303)
0618 22:20:18.319452 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0618 22:20:18.319457 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:20:18.320332 (+ 875us) server_negotiation.cc:407] Connection header received
0618 22:20:18.321285 (+ 953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:20:18.321289 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:20:18.321353 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:20:18.321453 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:20:18.322544 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:18.323067 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:18.324321 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:18.324648 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:18.324733 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:18.328049 (+ 3316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:20:18.328053 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:20:18.328055 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:20:18.328076 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:20:18.329526 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:18.330091 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:18.330096 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:18.330098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:18.330177 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:18.330523 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:18.330526 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:18.330528 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:18.330695 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:20:18.330808 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0618 22:20:18.331273 (+ 465us) server_negotiation.cc:299] Negotiation successful
0618 22:20:18.331392 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260618 22:20:18.332341 13140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:18.319753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34303 (local address 127.0.98.67:50959)
0618 22:20:18.320190 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:18.320241 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0618 22:20:18.321086 (+ 845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:20:18.321621 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:20:18.321627 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:20:18.321880 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:20:18.322403 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:18.322416 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:18.323192 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:18.323195 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:18.324170 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:18.324182 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:18.324942 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:18.324946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:18.325142 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:18.326526 (+ 1384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:20:18.326548 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:20:18.327890 (+ 1342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:20:18.329658 (+ 1768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:20:18.329664 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:20:18.329677 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:20:18.329994 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:20:18.330275 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:20:18.330278 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:20:18.330280 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:20:18.330422 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:20:18.330799 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:20:18.330805 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:20:18.331043 (+ 238us) client_negotiation.cc:769] Sending connection context
0618 22:20:18.331218 (+ 175us) client_negotiation.cc:240] Negotiation successful
0618 22:20:18.331502 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":131,"threads_started":1}
I20260618 22:20:18.333385 13138 heartbeater.cc:347] Connected to a master server at 127.0.98.126:34303
I20260618 22:20:18.333593 13138 heartbeater.cc:464] Registering TS with master...
I20260618 22:20:18.334089 13138 heartbeater.cc:511] Master 127.0.98.126:34303 requested a full tablet report, sending...
I20260618 22:20:18.335088 12654 ts_manager.cc:194] Registered new tserver with Master: 3ba34efe46904a9581b54eea2f6b6cda (127.0.98.67:35107)
I20260618 22:20:18.335628 12654 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.67@KRBTEST.COM'} at 127.0.98.67:50959
I20260618 22:20:18.338163 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 22:20:18.338773 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 12735
I20260618 22:20:18.344825 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 12874
I20260618 22:20:18.349944 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 13009
I20260618 22:20:18.355165 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 12622
I20260618 22:20:18.361466 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:34303
--webserver_interface=127.0.98.126
--webserver_port=34747
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:35427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:34303
--ranger_config_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.84:55441/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.84:55441/kms with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:20:18.468693 13141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:20:18.468930 13141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:20:18.469010 13141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:20:18.472980 13141 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:20:18.473034 13141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:20:18.473053 13141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:20:18.473114 13141 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:20:18.473147 13141 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:20:18.478462 13141 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:35427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.84:55441/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:34303
--ranger_config_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:34303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=34747
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:20:18.479420 13141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:20:18.480371 13141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:20:18.489027 13147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:20:18.489111 13146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:20:18.489099 13149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:20:18.489218 13141 server_base.cc:1034] running on GCE node
I20260618 22:20:18.489800 13141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:20:18.490882 13141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:20:18.492066 13141 hybrid_clock.cc:648] HybridClock initialized: now 1781821218492049 us; error 30 us; skew 500 ppm
Jun 18 22:20:18 dist-test-slave-g6r4 krb5kdc[12178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821218, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:20:18.494562 13141 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260618 22:20:18.495342 13141 webserver.cc:460] Webserver started at http://127.0.98.126:34747/ using document root <none> and password file <none>
I20260618 22:20:18.495837 13141 fs_manager.cc:359] Metadata directory not provided
I20260618 22:20:18.495923 13141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 18 22:20:18 dist-test-slave-g6r4 krb5kdc[12178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821218, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for HTTP/127.0.98.84@KRBTEST.COM
I20260618 22:20:18.543531 13141 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260618 22:20:18.546169 13157 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:20:18.547713 13141 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:20:18.547843 13141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/master-0/wal
uuid: "3096d8ecc01b437a8f0beabf8fb021f7"
format_stamp: "Formatted at 2026-06-18 22:20:16 on dist-test-slave-g6r4"
server_key: "d3df98ecd4b980cc7935d149ba34685e"
server_key_iv: "6029f6cc886ce2b9bd54b8114accc417"
server_key_version: "kuduclusterkey@0"
I20260618 22:20:18.548516 13141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:20:18.573210 13141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:20:18.576582 13141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:20:18.576776 13141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:20:18.584977 13141 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:34303
I20260618 22:20:18.584982 13209 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:34303 every 8 connection(s)
I20260618 22:20:18.586272 13141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
I20260618 22:20:18.588207 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 13141
I20260618 22:20:18.588696 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.84:55441/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:40163
--local_ip_for_outbound_sockets=127.0.98.65
--tserver_master_addrs=127.0.98.126:34303
--webserver_port=43881
--webserver_interface=127.0.98.65
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:35427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.84:55441/kms with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 22:20:18.590953 13210 sys_catalog.cc:263] Verifying existing consensus state
I20260618 22:20:18.592584 13210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3096d8ecc01b437a8f0beabf8fb021f7: Bootstrap starting.
I20260618 22:20:18.609124 13210 log.cc:824] T 00000000000000000000000000000000 P 3096d8ecc01b437a8f0beabf8fb021f7: Log is configured to *not* fsync() on all Append() calls
I20260618 22:20:18.616029 13210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3096d8ecc01b437a8f0beabf8fb021f7: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260618 22:20:18.616454 13210 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3096d8ecc01b437a8f0beabf8fb021f7: Bootstrap complete.
I20260618 22:20:18.620921 13210 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3096d8ecc01b437a8f0beabf8fb021f7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3096d8ecc01b437a8f0beabf8fb021f7" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34303 } }
I20260618 22:20:18.621316 13210 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3096d8ecc01b437a8f0beabf8fb021f7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3096d8ecc01b437a8f0beabf8fb021f7, State: Initialized, Role: FOLLOWER
I20260618 22:20:18.621870 13210 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3096d8ecc01b437a8f0beabf8fb021f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3096d8ecc01b437a8f0beabf8fb021f7" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34303 } }
I20260618 22:20:18.622088 13210 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3096d8ecc01b437a8f0beabf8fb021f7 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:20:18.622160 13210 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3096d8ecc01b437a8f0beabf8fb021f7 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:20:18.622262 13210 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3096d8ecc01b437a8f0beabf8fb021f7 [term 1 FOLLOWER]: Advancing to term 2
I20260618 22:20:18.624209 13210 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3096d8ecc01b437a8f0beabf8fb021f7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3096d8ecc01b437a8f0beabf8fb021f7" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34303 } }
I20260618 22:20:18.624559 13210 leader_election.cc:302] T 00000000000000000000000000000000 P 3096d8ecc01b437a8f0beabf8fb021f7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3096d8ecc01b437a8f0beabf8fb021f7; no voters:
I20260618 22:20:18.624884 13210 leader_election.cc:288] T 00000000000000000000000000000000 P 3096d8ecc01b437a8f0beabf8fb021f7 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260618 22:20:18.624995 13214 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3096d8ecc01b437a8f0beabf8fb021f7 [term 2 FOLLOWER]: Leader election won for term 2
I20260618 22:20:18.625248 13214 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3096d8ecc01b437a8f0beabf8fb021f7 [term 2 LEADER]: Becoming Leader. State: Replica: 3096d8ecc01b437a8f0beabf8fb021f7, State: Running, Role: LEADER
I20260618 22:20:18.625602 13214 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3096d8ecc01b437a8f0beabf8fb021f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3096d8ecc01b437a8f0beabf8fb021f7" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34303 } }
I20260618 22:20:18.626153 13210 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3096d8ecc01b437a8f0beabf8fb021f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:20:18.627002 13215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3096d8ecc01b437a8f0beabf8fb021f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "3096d8ecc01b437a8f0beabf8fb021f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3096d8ecc01b437a8f0beabf8fb021f7" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34303 } } }
I20260618 22:20:18.627141 13215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3096d8ecc01b437a8f0beabf8fb021f7 [sys.catalog]: This master's current role is: LEADER
I20260618 22:20:18.627525 13216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3096d8ecc01b437a8f0beabf8fb021f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3096d8ecc01b437a8f0beabf8fb021f7. Latest consensus state: current_term: 2 leader_uuid: "3096d8ecc01b437a8f0beabf8fb021f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3096d8ecc01b437a8f0beabf8fb021f7" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34303 } } }
I20260618 22:20:18.627673 13216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3096d8ecc01b437a8f0beabf8fb021f7 [sys.catalog]: This master's current role is: LEADER
I20260618 22:20:18.627794 13222 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:20:18.630963 13222 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:20:18.631510 13210 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260618 22:20:18.631655 13222 catalog_manager.cc:1259] Loaded cluster ID: 68b8d6d2b17547b4aec69dcf2be39c00
I20260618 22:20:18.631764 13222 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:20:18.634024 13222 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:20:18.637161 13222 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 3096d8ecc01b437a8f0beabf8fb021f7: Loaded TSK: 0
I20260618 22:20:18.638799 13222 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260618 22:20:18.724144 13212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:20:18.724572 13212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:20:18.724709 13212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:20:18.731249 13212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:20:18.731317 13212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:20:18.731402 13212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260618 22:20:18.739081 13212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:35427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.84:55441/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:40163
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=43881
--tserver_master_addrs=127.0.98.126:34303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:20:18.740388 13212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:20:18.741598 13212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:20:18.753409 13250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:20:18.753691 13248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:20:18.753520 13247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:20:18.753930 13212 server_base.cc:1034] running on GCE node
I20260618 22:20:18.754483 13212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:20:18.755215 13212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:20:18.756418 13212 hybrid_clock.cc:648] HybridClock initialized: now 1781821218756392 us; error 39 us; skew 500 ppm
Jun 18 22:20:18 dist-test-slave-g6r4 krb5kdc[12178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821218, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:20:18.760210 13212 init.cc:377] Logged in from keytab as kudu/127.0.98.65@KRBTEST.COM (short username kudu)
I20260618 22:20:18.761126 13212 webserver.cc:460] Webserver started at http://127.0.98.65:43881/ using document root <none> and password file <none>
I20260618 22:20:18.761672 13212 fs_manager.cc:359] Metadata directory not provided
I20260618 22:20:18.761775 13212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 18 22:20:18 dist-test-slave-g6r4 krb5kdc[12178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821218, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for HTTP/127.0.98.84@KRBTEST.COM
I20260618 22:20:18.782346 13212 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:20:18.784901 13258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:20:18.786253 13212 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:20:18.786379 13212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-0/wal
uuid: "7d0f3194f19141c697b072ecc2faea28"
format_stamp: "Formatted at 2026-06-18 22:20:17 on dist-test-slave-g6r4"
server_key: "0341ddf91565988f7e0934f7308d6731"
server_key_iv: "66ee6e273a49d646996a063baa9e8bd2"
server_key_version: "kuduclusterkey@0"
I20260618 22:20:18.786919 13212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:20:18.817231 13212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:20:18.820703 13212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:20:18.820973 13212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:20:18.821664 13212 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:20:18.822705 13212 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:20:18.822813 13212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:20:18.822897 13212 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:20:18.822943 13212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:20:18.834272 13212 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:40163
I20260618 22:20:18.835520 13212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
I20260618 22:20:18.835861 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 13212
I20260618 22:20:18.836380 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.84:55441/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:34817
--local_ip_for_outbound_sockets=127.0.98.66
--tserver_master_addrs=127.0.98.126:34303
--webserver_port=44021
--webserver_interface=127.0.98.66
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:35427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.84:55441/kms with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 22:20:18.842278 13371 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:40163 every 8 connection(s)
Jun 18 22:20:18 dist-test-slave-g6r4 krb5kdc[12178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821218, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:20:18.868094 13376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:18.843344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34303 (local address 127.0.98.65:58705)
0618 22:20:18.843840 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:18.843891 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0618 22:20:18.844888 (+ 997us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:20:18.846984 (+ 2096us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:20:18.846993 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:20:18.847333 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:20:18.847934 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:18.847952 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:18.849790 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:18.849794 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:18.851185 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:18.851195 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:18.852421 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:18.852425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:18.852613 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:18.853358 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:20:18.853380 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:20:18.859082 (+ 5702us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:20:18.864608 (+ 5526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:20:18.864614 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:20:18.864629 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:20:18.864962 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:20:18.865377 (+ 415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:20:18.865381 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:20:18.865383 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:20:18.865561 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:20:18.866303 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:20:18.866310 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:20:18.866626 (+ 316us) client_negotiation.cc:769] Sending connection context
0618 22:20:18.866807 (+ 181us) client_negotiation.cc:240] Negotiation successful
0618 22:20:18.867124 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":118,"threads_started":1}
I20260618 22:20:18.870128 13375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:18.843213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:58705 (local address 127.0.98.126:34303)
0618 22:20:18.845378 (+ 2165us) server_negotiation.cc:206] Beginning negotiation
0618 22:20:18.845400 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0618 22:20:18.845443 (+ 43us) server_negotiation.cc:407] Connection header received
0618 22:20:18.845731 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:20:18.845745 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:20:18.846174 (+ 429us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:20:18.846513 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:20:18.848116 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:18.849643 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:18.851329 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:18.852055 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:18.852214 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:18.862070 (+ 9856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:20:18.862079 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:20:18.862098 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:20:18.862128 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:20:18.864394 (+ 2266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:18.865111 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:18.865117 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:18.865122 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:18.865263 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:18.865686 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:18.865689 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:18.865691 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:18.866168 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:20:18.868822 (+ 2654us) server_negotiation.cc:1035] Waiting for connection context
0618 22:20:18.868966 (+ 144us) server_negotiation.cc:299] Negotiation successful
0618 22:20:18.869243 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2008,"thread_start_us":121,"threads_started":1}
I20260618 22:20:18.872587 13372 heartbeater.cc:347] Connected to a master server at 127.0.98.126:34303
I20260618 22:20:18.872817 13372 heartbeater.cc:464] Registering TS with master...
I20260618 22:20:18.873349 13372 heartbeater.cc:511] Master 127.0.98.126:34303 requested a full tablet report, sending...
W20260618 22:20:18.875468 13372 heartbeater.cc:644] Failed to heartbeat to 127.0.98.126:34303 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260618 22:20:18.875581 13372 heartbeater.cc:464] Registering TS with master...
I20260618 22:20:18.875741 13372 heartbeater.cc:511] Master 127.0.98.126:34303 requested a full tablet report, sending...
I20260618 22:20:18.876638 13372 heartbeater.cc:464] Registering TS with master...
I20260618 22:20:18.876794 13372 heartbeater.cc:511] Master 127.0.98.126:34303 requested a full tablet report, sending...
W20260618 22:20:18.877527 13372 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260618 22:20:18.997599 13374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:20:18.997962 13374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:20:18.998032 13374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:20:19.004160 13374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:20:19.004231 13374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:20:19.004318 13374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260618 22:20:19.011554 13374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:35427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.84:55441/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:34817
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=44021
--tserver_master_addrs=127.0.98.126:34303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:20:19.012578 13374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:20:19.013592 13374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:20:19.028728 13381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:20:19.028734 13382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:20:19.029191 13384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:20:19.029371 13374 server_base.cc:1034] running on GCE node
I20260618 22:20:19.029737 13374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:20:19.030463 13374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:20:19.031648 13374 hybrid_clock.cc:648] HybridClock initialized: now 1781821219031634 us; error 28 us; skew 500 ppm
Jun 18 22:20:19 dist-test-slave-g6r4 krb5kdc[12178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821219, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:20:19.035028 13374 init.cc:377] Logged in from keytab as kudu/127.0.98.66@KRBTEST.COM (short username kudu)
I20260618 22:20:19.035882 13374 webserver.cc:460] Webserver started at http://127.0.98.66:44021/ using document root <none> and password file <none>
I20260618 22:20:19.036422 13374 fs_manager.cc:359] Metadata directory not provided
I20260618 22:20:19.036526 13374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 18 22:20:19 dist-test-slave-g6r4 krb5kdc[12178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821219, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for HTTP/127.0.98.84@KRBTEST.COM
I20260618 22:20:19.052327 13374 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:20:19.055361 13392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:20:19.056727 13374 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:20:19.056833 13374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-1/wal
uuid: "4e4d9d282a65497e9189190a01a26bab"
format_stamp: "Formatted at 2026-06-18 22:20:17 on dist-test-slave-g6r4"
server_key: "9fe0a17760b6a96515761025c134aa74"
server_key_iv: "f718f66c93924d9f4eb1e8dcc952a43e"
server_key_version: "kuduclusterkey@0"
I20260618 22:20:19.057350 13374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:20:19.071277 13374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:20:19.074316 13374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:20:19.074519 13374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:20:19.075181 13374 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:20:19.076150 13374 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:20:19.076213 13374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:20:19.076277 13374 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:20:19.076308 13374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:20:19.087365 13374 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:34817
I20260618 22:20:19.088596 13374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
I20260618 22:20:19.095131 13505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:34817 every 8 connection(s)
I20260618 22:20:19.098064 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 13374
I20260618 22:20:19.098551 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.84:55441/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:35107
--local_ip_for_outbound_sockets=127.0.98.67
--tserver_master_addrs=127.0.98.126:34303
--webserver_port=38969
--webserver_interface=127.0.98.67
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:35427
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.84:55441/kms with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 18 22:20:19 dist-test-slave-g6r4 krb5kdc[12178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821219, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:20:19.114989 13375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:19.097273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:53159 (local address 127.0.98.126:34303)
0618 22:20:19.097438 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0618 22:20:19.097443 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:20:19.099375 (+ 1932us) server_negotiation.cc:407] Connection header received
0618 22:20:19.100785 (+ 1410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:20:19.100791 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:20:19.100854 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:20:19.100915 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:20:19.102883 (+ 1968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:19.103899 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:19.106111 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:19.106585 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:19.106694 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:19.110180 (+ 3486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:20:19.110186 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:20:19.110190 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:20:19.110227 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:20:19.112370 (+ 2143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:19.113052 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:19.113056 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:19.113058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:19.113149 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:19.113610 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:19.113613 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:19.113615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:19.113850 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:20:19.113967 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0618 22:20:19.114648 (+ 681us) server_negotiation.cc:299] Negotiation successful
0618 22:20:19.114811 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260618 22:20:19.116916 13508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:19.097720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34303 (local address 127.0.98.66:53159)
0618 22:20:19.099214 (+ 1494us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:19.099270 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0618 22:20:19.100518 (+ 1248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:20:19.101368 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:20:19.101377 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:20:19.101847 (+ 470us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:20:19.102648 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:19.102673 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:19.104098 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:19.104105 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:19.105940 (+ 1835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:19.105957 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:19.106958 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:19.106966 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:19.107226 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:19.107993 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:20:19.108023 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:20:19.109948 (+ 1925us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:20:19.112518 (+ 2570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:20:19.112525 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:20:19.112541 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:20:19.112916 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:20:19.113303 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:20:19.113306 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:20:19.113308 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:20:19.113487 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:20:19.114036 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:20:19.114042 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:20:19.114362 (+ 320us) client_negotiation.cc:769] Sending connection context
0618 22:20:19.115532 (+ 1170us) client_negotiation.cc:240] Negotiation successful
0618 22:20:19.115891 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1265,"thread_start_us":169,"threads_started":1}
I20260618 22:20:19.118428 13506 heartbeater.cc:347] Connected to a master server at 127.0.98.126:34303
I20260618 22:20:19.118673 13506 heartbeater.cc:464] Registering TS with master...
I20260618 22:20:19.119261 13506 heartbeater.cc:511] Master 127.0.98.126:34303 requested a full tablet report, sending...
W20260618 22:20:19.120817 13506 heartbeater.cc:644] Failed to heartbeat to 127.0.98.126:34303 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260618 22:20:19.120932 13506 heartbeater.cc:464] Registering TS with master...
I20260618 22:20:19.121080 13506 heartbeater.cc:511] Master 127.0.98.126:34303 requested a full tablet report, sending...
I20260618 22:20:19.121933 13506 heartbeater.cc:464] Registering TS with master...
I20260618 22:20:19.122104 13506 heartbeater.cc:511] Master 127.0.98.126:34303 requested a full tablet report, sending...
W20260618 22:20:19.122916 13506 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260618 22:20:19.279198 13509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:20:19.279623 13509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:20:19.279703 13509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:20:19.286701 13509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:20:19.286775 13509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:20:19.286873 13509 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260618 22:20:19.295480 13509 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:35427
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.84:55441/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:35107
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=38969
--tserver_master_addrs=127.0.98.126:34303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:20:19.296885 13509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:20:19.298305 13509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:20:19.311017 13515 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:20:19.311087 13516 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:20:19.311043 13518 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:20:19.311440 13509 server_base.cc:1034] running on GCE node
I20260618 22:20:19.311887 13509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:20:19.312709 13509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:20:19.313953 13509 hybrid_clock.cc:648] HybridClock initialized: now 1781821219313930 us; error 36 us; skew 500 ppm
Jun 18 22:20:19 dist-test-slave-g6r4 krb5kdc[12178](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821219, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:20:19.318151 13509 init.cc:377] Logged in from keytab as kudu/127.0.98.67@KRBTEST.COM (short username kudu)
I20260618 22:20:19.319361 13509 webserver.cc:460] Webserver started at http://127.0.98.67:38969/ using document root <none> and password file <none>
I20260618 22:20:19.320142 13509 fs_manager.cc:359] Metadata directory not provided
I20260618 22:20:19.320302 13509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 18 22:20:19 dist-test-slave-g6r4 krb5kdc[12178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821219, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for HTTP/127.0.98.84@KRBTEST.COM
I20260618 22:20:19.344852 13509 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:20:19.356782 13526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:20:19.358580 13509 fs_manager.cc:725] Time spent opening block manager: real 0.012s user 0.002s sys 0.000s
I20260618 22:20:19.358736 13509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-2/wal
uuid: "3ba34efe46904a9581b54eea2f6b6cda"
format_stamp: "Formatted at 2026-06-18 22:20:18 on dist-test-slave-g6r4"
server_key: "838a0633ae8c5ba5d36a6d8c58571d1c"
server_key_iv: "ad8a9cf7e2ec5e6daa66bb90867dd193"
server_key_version: "kuduclusterkey@0"
I20260618 22:20:19.359500 13509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:20:19.377480 13509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:20:19.381347 13509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:20:19.381556 13509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:20:19.382328 13509 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:20:19.383399 13509 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:20:19.383472 13509 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:20:19.383538 13509 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:20:19.383575 13509 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:20:19.396242 13509 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:35107
I20260618 22:20:19.397604 13509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
I20260618 22:20:19.399966 13639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:35107 every 8 connection(s)
I20260618 22:20:19.402091 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 13509
Jun 18 22:20:19 dist-test-slave-g6r4 krb5kdc[12178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821219, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:20:19.426695 13644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:19.403561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48944 (local address 127.0.98.126:34303)
0618 22:20:19.403835 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0618 22:20:19.403840 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:20:19.410220 (+ 6380us) server_negotiation.cc:407] Connection header received
0618 22:20:19.410348 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:20:19.410353 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:20:19.410419 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:20:19.410531 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:20:19.412741 (+ 2210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:19.413551 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:19.415126 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:19.415467 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:19.415562 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:19.417417 (+ 1855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:20:19.417421 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:20:19.417424 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:20:19.417452 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:20:19.419331 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:19.420107 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:19.420111 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:19.420113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:19.420200 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:19.420536 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:19.420542 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:19.420544 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:19.420816 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:20:19.421039 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0618 22:20:19.426018 (+ 4979us) server_negotiation.cc:299] Negotiation successful
0618 22:20:19.426429 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":86,"threads_started":1}
I20260618 22:20:19.430207 13375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:19.401773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:34093 (local address 127.0.98.126:34303)
0618 22:20:19.402549 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0618 22:20:19.402554 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:20:19.410739 (+ 8185us) server_negotiation.cc:407] Connection header received
0618 22:20:19.411897 (+ 1158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:20:19.411900 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:20:19.411965 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:20:19.412059 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:20:19.415419 (+ 3360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:19.416268 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:19.418136 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:19.418524 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:19.418607 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:19.425123 (+ 6516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:20:19.425127 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:20:19.425130 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:20:19.425157 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:20:19.427111 (+ 1954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:19.428165 (+ 1054us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:19.428169 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:19.428171 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:19.428263 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:19.428779 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:19.428783 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:19.428785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:19.429024 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:20:19.429120 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0618 22:20:19.429887 (+ 767us) server_negotiation.cc:299] Negotiation successful
0618 22:20:19.430046 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":680}
I20260618 22:20:19.432184 13642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:19.402344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34303 (local address 127.0.98.67:34093)
0618 22:20:19.410571 (+ 8227us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:19.410628 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0618 22:20:19.411649 (+ 1021us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:20:19.412366 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:20:19.412376 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:20:19.412786 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:20:19.415228 (+ 2442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:19.415251 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:19.416436 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:19.416439 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:19.417991 (+ 1552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:19.418002 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:19.418867 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:19.418871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:19.419098 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:19.419719 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:20:19.419742 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:20:19.424909 (+ 5167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:20:19.427594 (+ 2685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:20:19.427602 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:20:19.427620 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:20:19.428030 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:20:19.428426 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:20:19.428430 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:20:19.428432 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:20:19.428642 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:20:19.429195 (+ 553us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:20:19.429202 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:20:19.429563 (+ 361us) client_negotiation.cc:769] Sending connection context
0618 22:20:19.430697 (+ 1134us) client_negotiation.cc:240] Negotiation successful
0618 22:20:19.431067 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8009,"thread_start_us":144,"threads_started":1}
I20260618 22:20:19.433641 13640 heartbeater.cc:347] Connected to a master server at 127.0.98.126:34303
I20260618 22:20:19.433888 13640 heartbeater.cc:464] Registering TS with master...
I20260618 22:20:19.434516 13640 heartbeater.cc:511] Master 127.0.98.126:34303 requested a full tablet report, sending...
W20260618 22:20:19.436148 13640 heartbeater.cc:644] Failed to heartbeat to 127.0.98.126:34303 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260618 22:20:19.436276 13640 heartbeater.cc:464] Registering TS with master...
I20260618 22:20:19.436450 13640 heartbeater.cc:511] Master 127.0.98.126:34303 requested a full tablet report, sending...
I20260618 22:20:19.437320 13640 heartbeater.cc:464] Registering TS with master...
I20260618 22:20:19.437492 13640 heartbeater.cc:511] Master 127.0.98.126:34303 requested a full tablet report, sending...
W20260618 22:20:19.438340 13640 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 18 22:20:19 dist-test-slave-g6r4 krb5kdc[12178](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781821219, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:20:19.877767 13372 heartbeater.cc:464] Registering TS with master...
I20260618 22:20:19.878127 13372 heartbeater.cc:511] Master 127.0.98.126:34303 requested a full tablet report, sending...
I20260618 22:20:19.879204 13174 ts_manager.cc:194] Registered new tserver with Master: 7d0f3194f19141c697b072ecc2faea28 (127.0.98.65:40163)
I20260618 22:20:19.880811 13174 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.65@KRBTEST.COM'} at 127.0.98.65:58705
I20260618 22:20:20.123137 13506 heartbeater.cc:464] Registering TS with master...
I20260618 22:20:20.123436 13506 heartbeater.cc:511] Master 127.0.98.126:34303 requested a full tablet report, sending...
I20260618 22:20:20.124135 13174 ts_manager.cc:194] Registered new tserver with Master: 4e4d9d282a65497e9189190a01a26bab (127.0.98.66:34817)
I20260618 22:20:20.124907 13174 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.66@KRBTEST.COM'} at 127.0.98.66:53159
Jun 18 22:20:20 dist-test-slave-g6r4 krb5kdc[12178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821219, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for HTTP/127.0.98.84@KRBTEST.COM
I20260618 22:20:20.438562 13640 heartbeater.cc:464] Registering TS with master...
I20260618 22:20:20.438854 13640 heartbeater.cc:511] Master 127.0.98.126:34303 requested a full tablet report, sending...
I20260618 22:20:20.439600 13174 ts_manager.cc:194] Registered new tserver with Master: 3ba34efe46904a9581b54eea2f6b6cda (127.0.98.67:35107)
I20260618 22:20:20.440399 13174 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.67@KRBTEST.COM'} at 127.0.98.67:34093
I20260618 22:20:20.445675 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 22:20:20.459280 13661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:20.447408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48954 (local address 127.0.98.126:34303)
0618 22:20:20.450072 (+ 2664us) server_negotiation.cc:206] Beginning negotiation
0618 22:20:20.450078 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:20:20.450092 (+ 14us) server_negotiation.cc:407] Connection header received
0618 22:20:20.450130 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:20:20.450133 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:20:20.450190 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:20:20.450285 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:20:20.451177 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:20.452002 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:20.453197 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:20.453570 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:20.453649 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:20.455483 (+ 1834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:20:20.455486 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:20:20.455489 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:20:20.455514 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:20:20.457300 (+ 1786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:20.457798 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:20.457802 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:20.457804 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:20.457889 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:20.458246 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:20.458250 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:20.458251 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:20.458470 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:20:20.458579 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0618 22:20:20.458953 (+ 374us) server_negotiation.cc:299] Negotiation successful
0618 22:20:20.459082 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2587,"thread_start_us":85,"threads_started":1}
I20260618 22:20:20.463976 13174 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48954:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 22:20:20.469612 13174 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
Jun 18 22:20:20 dist-test-slave-g6r4 krb5kdc[12178](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821219, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for HTTP/127.0.98.84@KRBTEST.COM
I20260618 22:20:20.507431 13671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:20.499198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:34817 (local address 127.0.0.1:59418)
0618 22:20:20.499969 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:20.499981 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 22:20:20.500048 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:20:20.501320 (+ 1272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:20:20.501326 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:20:20.501352 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:20:20.501498 (+ 146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:20.501507 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:20.503668 (+ 2161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:20.503672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:20.505086 (+ 1414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:20.505097 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:20.506904 (+ 1807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:20.506908 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:20.507024 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:20.507065 (+ 41us) client_negotiation.cc:769] Sending connection context
0618 22:20:20.507149 (+ 84us) client_negotiation.cc:240] Negotiation successful
0618 22:20:20.507273 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":714,"thread_start_us":99,"threads_started":1}
I20260618 22:20:20.508405 13672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:20.500009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59418 (local address 127.0.98.66:34817)
0618 22:20:20.500457 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0618 22:20:20.500465 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0618 22:20:20.500491 (+ 26us) server_negotiation.cc:407] Connection header received
0618 22:20:20.500931 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:20:20.500937 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:20:20.501139 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:20:20.501302 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:20:20.501632 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:20.502862 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:20.505258 (+ 2396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:20.506772 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:20.507963 (+ 1191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:20.508106 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0618 22:20:20.508159 (+ 53us) server_negotiation.cc:299] Negotiation successful
0618 22:20:20.508266 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":113,"threads_started":1}
I20260618 22:20:20.511425 13436 tablet_service.cc:1467] Processing CreateTablet for tablet cfff7f3301b3420db337ede054013b10 (DEFAULT_TABLE table=test-table [id=db8f4b9b9c034c3bae2630b4b6b77f24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:20:20.512542 13436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfff7f3301b3420db337ede054013b10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:20:20.514868 13670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:20.498462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:40163 (local address 127.0.0.1:60850)
0618 22:20:20.499586 (+ 1124us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:20.499634 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0618 22:20:20.499754 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:20:20.507884 (+ 8130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:20:20.507887 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:20:20.507907 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:20:20.507995 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:20.508001 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:20.512065 (+ 4064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:20.512069 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:20.513263 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:20.513274 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:20.514482 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:20.514486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:20.514588 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:20.514614 (+ 26us) client_negotiation.cc:769] Sending connection context
0618 22:20:20.514668 (+ 54us) client_negotiation.cc:240] Negotiation successful
0618 22:20:20.514734 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":947,"thread_start_us":102,"threads_started":1}
I20260618 22:20:20.515707 13673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:20.500159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60850 (local address 127.0.98.65:40163)
0618 22:20:20.506250 (+ 6091us) server_negotiation.cc:206] Beginning negotiation
0618 22:20:20.506257 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:20:20.506282 (+ 25us) server_negotiation.cc:407] Connection header received
0618 22:20:20.506357 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:20:20.506362 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:20:20.506557 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:20:20.508149 (+ 1592us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:20:20.508325 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:20.509714 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:20.513498 (+ 3784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:20.514341 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:20.515295 (+ 954us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:20.515436 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0618 22:20:20.515485 (+ 49us) server_negotiation.cc:299] Negotiation successful
0618 22:20:20.515569 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5973,"thread_start_us":73,"threads_started":1}
I20260618 22:20:20.520987 13306 tablet_service.cc:1467] Processing CreateTablet for tablet cfff7f3301b3420db337ede054013b10 (DEFAULT_TABLE table=test-table [id=db8f4b9b9c034c3bae2630b4b6b77f24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:20:20.521849 13306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfff7f3301b3420db337ede054013b10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:20:20.529028 13671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:20.499524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:35107 (local address 127.0.0.1:50600)
0618 22:20:20.507589 (+ 8065us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:20.507602 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:20:20.507675 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:20:20.522426 (+ 14751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:20:20.522429 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:20:20.522446 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:20:20.522528 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:20.522533 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:20.525015 (+ 2482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:20.525018 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:20.526249 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:20.526260 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:20.528632 (+ 2372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:20.528635 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:20.528745 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:20.528771 (+ 26us) client_negotiation.cc:769] Sending connection context
0618 22:20:20.528826 (+ 55us) client_negotiation.cc:240] Negotiation successful
0618 22:20:20.528892 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8015,"thread_start_us":81,"threads_started":1}
I20260618 22:20:20.529656 13674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:20.500455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50600 (local address 127.0.98.67:35107)
0618 22:20:20.500696 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0618 22:20:20.500702 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:20:20.521969 (+ 21267us) server_negotiation.cc:407] Connection header received
0618 22:20:20.522061 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:20:20.522066 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:20:20.522268 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:20:20.522674 (+ 406us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:20:20.522951 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:20.524890 (+ 1939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:20.526409 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:20.527842 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:20.529249 (+ 1407us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:20.529384 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0618 22:20:20.529433 (+ 49us) server_negotiation.cc:299] Negotiation successful
0618 22:20:20.529517 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":90,"threads_started":1}
I20260618 22:20:20.530133 13676 tablet_bootstrap.cc:492] T cfff7f3301b3420db337ede054013b10 P 7d0f3194f19141c697b072ecc2faea28: Bootstrap starting.
I20260618 22:20:20.530133 13675 tablet_bootstrap.cc:492] T cfff7f3301b3420db337ede054013b10 P 4e4d9d282a65497e9189190a01a26bab: Bootstrap starting.
I20260618 22:20:20.532392 13676 tablet_bootstrap.cc:654] T cfff7f3301b3420db337ede054013b10 P 7d0f3194f19141c697b072ecc2faea28: Neither blocks nor log segments found. Creating new log.
I20260618 22:20:20.532451 13574 tablet_service.cc:1467] Processing CreateTablet for tablet cfff7f3301b3420db337ede054013b10 (DEFAULT_TABLE table=test-table [id=db8f4b9b9c034c3bae2630b4b6b77f24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:20:20.532955 13675 tablet_bootstrap.cc:654] T cfff7f3301b3420db337ede054013b10 P 4e4d9d282a65497e9189190a01a26bab: Neither blocks nor log segments found. Creating new log.
I20260618 22:20:20.533241 13676 log.cc:824] T cfff7f3301b3420db337ede054013b10 P 7d0f3194f19141c697b072ecc2faea28: Log is configured to *not* fsync() on all Append() calls
I20260618 22:20:20.533428 13574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfff7f3301b3420db337ede054013b10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:20:20.533723 13675 log.cc:824] T cfff7f3301b3420db337ede054013b10 P 4e4d9d282a65497e9189190a01a26bab: Log is configured to *not* fsync() on all Append() calls
I20260618 22:20:20.535738 13676 tablet_bootstrap.cc:492] T cfff7f3301b3420db337ede054013b10 P 7d0f3194f19141c697b072ecc2faea28: No bootstrap required, opened a new log
I20260618 22:20:20.535934 13675 tablet_bootstrap.cc:492] T cfff7f3301b3420db337ede054013b10 P 4e4d9d282a65497e9189190a01a26bab: No bootstrap required, opened a new log
I20260618 22:20:20.535935 13676 ts_tablet_manager.cc:1397] T cfff7f3301b3420db337ede054013b10 P 7d0f3194f19141c697b072ecc2faea28: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260618 22:20:20.536072 13675 ts_tablet_manager.cc:1397] T cfff7f3301b3420db337ede054013b10 P 4e4d9d282a65497e9189190a01a26bab: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260618 22:20:20.539801 13676 raft_consensus.cc:348] T cfff7f3301b3420db337ede054013b10 P 7d0f3194f19141c697b072ecc2faea28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d0f3194f19141c697b072ecc2faea28" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 40163 } } peers { permanent_uuid: "4e4d9d282a65497e9189190a01a26bab" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 34817 } } peers { permanent_uuid: "3ba34efe46904a9581b54eea2f6b6cda" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 35107 } }
I20260618 22:20:20.540055 13676 raft_consensus.cc:374] T cfff7f3301b3420db337ede054013b10 P 7d0f3194f19141c697b072ecc2faea28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:20:20.540109 13676 raft_consensus.cc:729] T cfff7f3301b3420db337ede054013b10 P 7d0f3194f19141c697b072ecc2faea28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d0f3194f19141c697b072ecc2faea28, State: Initialized, Role: FOLLOWER
I20260618 22:20:20.540575 13676 consensus_queue.cc:260] T cfff7f3301b3420db337ede054013b10 P 7d0f3194f19141c697b072ecc2faea28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d0f3194f19141c697b072ecc2faea28" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 40163 } } peers { permanent_uuid: "4e4d9d282a65497e9189190a01a26bab" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 34817 } } peers { permanent_uuid: "3ba34efe46904a9581b54eea2f6b6cda" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 35107 } }
I20260618 22:20:20.540521 13675 raft_consensus.cc:348] T cfff7f3301b3420db337ede054013b10 P 4e4d9d282a65497e9189190a01a26bab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d0f3194f19141c697b072ecc2faea28" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 40163 } } peers { permanent_uuid: "4e4d9d282a65497e9189190a01a26bab" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 34817 } } peers { permanent_uuid: "3ba34efe46904a9581b54eea2f6b6cda" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 35107 } }
I20260618 22:20:20.540738 13675 raft_consensus.cc:374] T cfff7f3301b3420db337ede054013b10 P 4e4d9d282a65497e9189190a01a26bab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:20:20.540799 13675 raft_consensus.cc:729] T cfff7f3301b3420db337ede054013b10 P 4e4d9d282a65497e9189190a01a26bab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e4d9d282a65497e9189190a01a26bab, State: Initialized, Role: FOLLOWER
I20260618 22:20:20.541256 13675 consensus_queue.cc:260] T cfff7f3301b3420db337ede054013b10 P 4e4d9d282a65497e9189190a01a26bab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d0f3194f19141c697b072ecc2faea28" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 40163 } } peers { permanent_uuid: "4e4d9d282a65497e9189190a01a26bab" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 34817 } } peers { permanent_uuid: "3ba34efe46904a9581b54eea2f6b6cda" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 35107 } }
I20260618 22:20:20.541567 13676 ts_tablet_manager.cc:1428] T cfff7f3301b3420db337ede054013b10 P 7d0f3194f19141c697b072ecc2faea28: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260618 22:20:20.541930 13372 heartbeater.cc:503] Master 127.0.98.126:34303 was elected leader, sending a full tablet report...
I20260618 22:20:20.542251 13675 ts_tablet_manager.cc:1428] T cfff7f3301b3420db337ede054013b10 P 4e4d9d282a65497e9189190a01a26bab: Time spent starting tablet: real 0.006s user 0.005s sys 0.002s
I20260618 22:20:20.542548 13506 heartbeater.cc:503] Master 127.0.98.126:34303 was elected leader, sending a full tablet report...
I20260618 22:20:20.546377 13682 tablet_bootstrap.cc:492] T cfff7f3301b3420db337ede054013b10 P 3ba34efe46904a9581b54eea2f6b6cda: Bootstrap starting.
I20260618 22:20:20.548390 13682 tablet_bootstrap.cc:654] T cfff7f3301b3420db337ede054013b10 P 3ba34efe46904a9581b54eea2f6b6cda: Neither blocks nor log segments found. Creating new log.
I20260618 22:20:20.549088 13682 log.cc:824] T cfff7f3301b3420db337ede054013b10 P 3ba34efe46904a9581b54eea2f6b6cda: Log is configured to *not* fsync() on all Append() calls
I20260618 22:20:20.551116 13682 tablet_bootstrap.cc:492] T cfff7f3301b3420db337ede054013b10 P 3ba34efe46904a9581b54eea2f6b6cda: No bootstrap required, opened a new log
I20260618 22:20:20.551308 13682 ts_tablet_manager.cc:1397] T cfff7f3301b3420db337ede054013b10 P 3ba34efe46904a9581b54eea2f6b6cda: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260618 22:20:20.554983 13682 raft_consensus.cc:348] T cfff7f3301b3420db337ede054013b10 P 3ba34efe46904a9581b54eea2f6b6cda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d0f3194f19141c697b072ecc2faea28" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 40163 } } peers { permanent_uuid: "4e4d9d282a65497e9189190a01a26bab" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 34817 } } peers { permanent_uuid: "3ba34efe46904a9581b54eea2f6b6cda" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 35107 } }
I20260618 22:20:20.555255 13682 raft_consensus.cc:374] T cfff7f3301b3420db337ede054013b10 P 3ba34efe46904a9581b54eea2f6b6cda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:20:20.555312 13682 raft_consensus.cc:729] T cfff7f3301b3420db337ede054013b10 P 3ba34efe46904a9581b54eea2f6b6cda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ba34efe46904a9581b54eea2f6b6cda, State: Initialized, Role: FOLLOWER
I20260618 22:20:20.555745 13682 consensus_queue.cc:260] T cfff7f3301b3420db337ede054013b10 P 3ba34efe46904a9581b54eea2f6b6cda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d0f3194f19141c697b072ecc2faea28" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 40163 } } peers { permanent_uuid: "4e4d9d282a65497e9189190a01a26bab" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 34817 } } peers { permanent_uuid: "3ba34efe46904a9581b54eea2f6b6cda" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 35107 } }
I20260618 22:20:20.556398 13640 heartbeater.cc:503] Master 127.0.98.126:34303 was elected leader, sending a full tablet report...
I20260618 22:20:20.556728 13682 ts_tablet_manager.cc:1428] T cfff7f3301b3420db337ede054013b10 P 3ba34efe46904a9581b54eea2f6b6cda: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260618 22:20:20.594295 13373 tablet.cc:2369] T cfff7f3301b3420db337ede054013b10 P 7d0f3194f19141c697b072ecc2faea28: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260618 22:20:20.596715 13507 tablet.cc:2369] T cfff7f3301b3420db337ede054013b10 P 4e4d9d282a65497e9189190a01a26bab: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260618 22:20:20.652789 13641 tablet.cc:2369] T cfff7f3301b3420db337ede054013b10 P 3ba34efe46904a9581b54eea2f6b6cda: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:20:20.727839 13681 raft_consensus.cc:482] T cfff7f3301b3420db337ede054013b10 P 4e4d9d282a65497e9189190a01a26bab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:20:20.728087 13681 raft_consensus.cc:504] T cfff7f3301b3420db337ede054013b10 P 4e4d9d282a65497e9189190a01a26bab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d0f3194f19141c697b072ecc2faea28" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 40163 } } peers { permanent_uuid: "4e4d9d282a65497e9189190a01a26bab" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 34817 } } peers { permanent_uuid: "3ba34efe46904a9581b54eea2f6b6cda" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 35107 } }
I20260618 22:20:20.729395 13681 leader_election.cc:288] T cfff7f3301b3420db337ede054013b10 P 4e4d9d282a65497e9189190a01a26bab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d0f3194f19141c697b072ecc2faea28 (127.0.98.65:40163), 3ba34efe46904a9581b54eea2f6b6cda (127.0.98.67:35107)
I20260618 22:20:20.734067 13673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:20.729757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:56803 (local address 127.0.98.65:40163)
0618 22:20:20.729918 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0618 22:20:20.729923 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:20:20.730016 (+ 93us) server_negotiation.cc:407] Connection header received
0618 22:20:20.730169 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:20:20.730173 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:20:20.730252 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:20:20.730354 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:20:20.730818 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:20.731536 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:20.732808 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:20.733349 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:20.733447 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:20.733506 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0618 22:20:20.733820 (+ 314us) server_negotiation.cc:299] Negotiation successful
0618 22:20:20.733947 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260618 22:20:20.734477 13692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:20.729591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:40163 (local address 127.0.98.66:56803)
0618 22:20:20.729918 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:20.729934 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 22:20:20.730048 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:20:20.730509 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:20:20.730512 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:20:20.730530 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:20:20.730644 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:20.730649 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:20.731671 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:20.731674 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:20.732685 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:20.732694 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:20.733563 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:20.733566 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:20.733675 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:20.733704 (+ 29us) client_negotiation.cc:769] Sending connection context
0618 22:20:20.734223 (+ 519us) client_negotiation.cc:240] Negotiation successful
0618 22:20:20.734291 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"mutex_wait_us":24,"thread_start_us":145,"threads_started":1}
I20260618 22:20:20.734882 13674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:20.730323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:50781 (local address 127.0.98.67:35107)
0618 22:20:20.730459 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0618 22:20:20.730463 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:20:20.730712 (+ 249us) server_negotiation.cc:407] Connection header received
0618 22:20:20.730818 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:20:20.730821 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:20:20.730882 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:20:20.730955 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:20:20.731239 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:20.731950 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:20.733504 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:20.734128 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:20.734217 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:20.734272 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0618 22:20:20.734690 (+ 418us) server_negotiation.cc:299] Negotiation successful
0618 22:20:20.734768 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260618 22:20:20.735176 13326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfff7f3301b3420db337ede054013b10" candidate_uuid: "4e4d9d282a65497e9189190a01a26bab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d0f3194f19141c697b072ecc2faea28" is_pre_election: true
I20260618 22:20:20.735276 13693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:20.730065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:35107 (local address 127.0.98.66:50781)
0618 22:20:20.730604 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:20.730618 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:20:20.730719 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:20:20.730980 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:20:20.730983 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:20:20.730999 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:20:20.731121 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:20.731127 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:20.732066 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:20.732070 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:20.733359 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:20.733370 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:20.734427 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:20.734431 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:20.734535 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:20.734561 (+ 26us) client_negotiation.cc:769] Sending connection context
0618 22:20:20.735065 (+ 504us) client_negotiation.cc:240] Negotiation successful
0618 22:20:20.735132 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"mutex_wait_us":26,"thread_start_us":87,"threads_started":1}
I20260618 22:20:20.735497 13326 raft_consensus.cc:2454] T cfff7f3301b3420db337ede054013b10 P 7d0f3194f19141c697b072ecc2faea28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e4d9d282a65497e9189190a01a26bab in term 0.
I20260618 22:20:20.735857 13594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfff7f3301b3420db337ede054013b10" candidate_uuid: "4e4d9d282a65497e9189190a01a26bab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ba34efe46904a9581b54eea2f6b6cda" is_pre_election: true
I20260618 22:20:20.735991 13393 leader_election.cc:302] T cfff7f3301b3420db337ede054013b10 P 4e4d9d282a65497e9189190a01a26bab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e4d9d282a65497e9189190a01a26bab, 7d0f3194f19141c697b072ecc2faea28; no voters:
I20260618 22:20:20.736120 13594 raft_consensus.cc:2454] T cfff7f3301b3420db337ede054013b10 P 3ba34efe46904a9581b54eea2f6b6cda [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4e4d9d282a65497e9189190a01a26bab in term 0.
I20260618 22:20:20.736213 13681 raft_consensus.cc:2781] T cfff7f3301b3420db337ede054013b10 P 4e4d9d282a65497e9189190a01a26bab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:20:20.736289 13681 raft_consensus.cc:482] T cfff7f3301b3420db337ede054013b10 P 4e4d9d282a65497e9189190a01a26bab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:20:20.736320 13681 raft_consensus.cc:3037] T cfff7f3301b3420db337ede054013b10 P 4e4d9d282a65497e9189190a01a26bab [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:20:20.737422 13681 raft_consensus.cc:504] T cfff7f3301b3420db337ede054013b10 P 4e4d9d282a65497e9189190a01a26bab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d0f3194f19141c697b072ecc2faea28" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 40163 } } peers { permanent_uuid: "4e4d9d282a65497e9189190a01a26bab" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 34817 } } peers { permanent_uuid: "3ba34efe46904a9581b54eea2f6b6cda" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 35107 } }
I20260618 22:20:20.737754 13681 leader_election.cc:288] T cfff7f3301b3420db337ede054013b10 P 4e4d9d282a65497e9189190a01a26bab [CANDIDATE]: Term 1 election: Requested vote from peers 7d0f3194f19141c697b072ecc2faea28 (127.0.98.65:40163), 3ba34efe46904a9581b54eea2f6b6cda (127.0.98.67:35107)
I20260618 22:20:20.738070 13326 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfff7f3301b3420db337ede054013b10" candidate_uuid: "4e4d9d282a65497e9189190a01a26bab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d0f3194f19141c697b072ecc2faea28"
I20260618 22:20:20.738147 13594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfff7f3301b3420db337ede054013b10" candidate_uuid: "4e4d9d282a65497e9189190a01a26bab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ba34efe46904a9581b54eea2f6b6cda"
I20260618 22:20:20.738195 13326 raft_consensus.cc:3037] T cfff7f3301b3420db337ede054013b10 P 7d0f3194f19141c697b072ecc2faea28 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:20:20.738272 13594 raft_consensus.cc:3037] T cfff7f3301b3420db337ede054013b10 P 3ba34efe46904a9581b54eea2f6b6cda [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:20:20.739173 13326 raft_consensus.cc:2454] T cfff7f3301b3420db337ede054013b10 P 7d0f3194f19141c697b072ecc2faea28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e4d9d282a65497e9189190a01a26bab in term 1.
I20260618 22:20:20.739460 13393 leader_election.cc:302] T cfff7f3301b3420db337ede054013b10 P 4e4d9d282a65497e9189190a01a26bab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e4d9d282a65497e9189190a01a26bab, 7d0f3194f19141c697b072ecc2faea28; no voters:
I20260618 22:20:20.739526 13594 raft_consensus.cc:2454] T cfff7f3301b3420db337ede054013b10 P 3ba34efe46904a9581b54eea2f6b6cda [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e4d9d282a65497e9189190a01a26bab in term 1.
I20260618 22:20:20.739650 13681 raft_consensus.cc:2781] T cfff7f3301b3420db337ede054013b10 P 4e4d9d282a65497e9189190a01a26bab [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:20:20.739874 13681 raft_consensus.cc:686] T cfff7f3301b3420db337ede054013b10 P 4e4d9d282a65497e9189190a01a26bab [term 1 LEADER]: Becoming Leader. State: Replica: 4e4d9d282a65497e9189190a01a26bab, State: Running, Role: LEADER
I20260618 22:20:20.740187 13681 consensus_queue.cc:237] T cfff7f3301b3420db337ede054013b10 P 4e4d9d282a65497e9189190a01a26bab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d0f3194f19141c697b072ecc2faea28" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 40163 } } peers { permanent_uuid: "4e4d9d282a65497e9189190a01a26bab" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 34817 } } peers { permanent_uuid: "3ba34efe46904a9581b54eea2f6b6cda" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 35107 } }
I20260618 22:20:20.743592 13174 catalog_manager.cc:5495] T cfff7f3301b3420db337ede054013b10 P 4e4d9d282a65497e9189190a01a26bab reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e4d9d282a65497e9189190a01a26bab (127.0.98.66). New cstate: current_term: 1 leader_uuid: "4e4d9d282a65497e9189190a01a26bab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d0f3194f19141c697b072ecc2faea28" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 40163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e4d9d282a65497e9189190a01a26bab" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 34817 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3ba34efe46904a9581b54eea2f6b6cda" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 35107 } health_report { overall_health: UNKNOWN } } }
I20260618 22:20:20.800093 13672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:20.796048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59426 (local address 127.0.98.66:34817)
0618 22:20:20.796185 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0618 22:20:20.796190 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:20:20.796241 (+ 51us) server_negotiation.cc:407] Connection header received
0618 22:20:20.796327 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:20:20.796330 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:20:20.796398 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:20:20.796511 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:20:20.796782 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:20.797528 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:20.798391 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:20.798712 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:20.798819 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:20.799127 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:20:20.799657 (+ 530us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:20:20.799785 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0618 22:20:20.799883 (+ 98us) server_negotiation.cc:299] Negotiation successful
0618 22:20:20.799940 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260618 22:20:20.806347 13326 raft_consensus.cc:1261] T cfff7f3301b3420db337ede054013b10 P 7d0f3194f19141c697b072ecc2faea28 [term 1 FOLLOWER]: Refusing update from remote peer 4e4d9d282a65497e9189190a01a26bab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:20:20.806438 13594 raft_consensus.cc:1261] T cfff7f3301b3420db337ede054013b10 P 3ba34efe46904a9581b54eea2f6b6cda [term 1 FOLLOWER]: Refusing update from remote peer 4e4d9d282a65497e9189190a01a26bab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:20:20.807076 13681 consensus_queue.cc:1035] T cfff7f3301b3420db337ede054013b10 P 4e4d9d282a65497e9189190a01a26bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d0f3194f19141c697b072ecc2faea28" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 40163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:20:20.807319 13694 consensus_queue.cc:1035] T cfff7f3301b3420db337ede054013b10 P 4e4d9d282a65497e9189190a01a26bab [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ba34efe46904a9581b54eea2f6b6cda" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 35107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:20:20.814262 13697 mvcc.cc:205] Tried to move back new op lower bound from 7298339720415666176 to 7298339720154660864. Current Snapshot: MvccSnapshot[applied={T|T < 7298339720415666176}]
I20260618 22:20:20.817076 13698 mvcc.cc:205] Tried to move back new op lower bound from 7298339720415666176 to 7298339720154660864. Current Snapshot: MvccSnapshot[applied={T|T < 7298339720415666176}]
I20260618 22:20:20.825507 13174 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:20:20.825704 13174 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48954:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:20:20.828236 13174 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3096d8ecc01b437a8f0beabf8fb021f7: Sending DeleteTablet for 3 replicas of tablet cfff7f3301b3420db337ede054013b10
I20260618 22:20:20.829217 13436 tablet_service.cc:1514] Processing DeleteTablet for tablet cfff7f3301b3420db337ede054013b10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:20:20 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:59418
I20260618 22:20:20.829231 13574 tablet_service.cc:1514] Processing DeleteTablet for tablet cfff7f3301b3420db337ede054013b10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:20:20 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:50600
I20260618 22:20:20.829850 13306 tablet_service.cc:1514] Processing DeleteTablet for tablet cfff7f3301b3420db337ede054013b10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:20:20 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:60850
I20260618 22:20:20.830341 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 13212
I20260618 22:20:20.838147 13707 tablet_replica.cc:330] T cfff7f3301b3420db337ede054013b10 P 4e4d9d282a65497e9189190a01a26bab: stopping tablet replica
I20260618 22:20:20.838469 13707 raft_consensus.cc:2229] T cfff7f3301b3420db337ede054013b10 P 4e4d9d282a65497e9189190a01a26bab [term 1 LEADER]: Raft consensus shutting down.
I20260618 22:20:20.838575 13706 tablet_replica.cc:330] T cfff7f3301b3420db337ede054013b10 P 3ba34efe46904a9581b54eea2f6b6cda: stopping tablet replica
I20260618 22:20:20.838815 13707 raft_consensus.cc:2258] T cfff7f3301b3420db337ede054013b10 P 4e4d9d282a65497e9189190a01a26bab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:20:20.838832 13706 raft_consensus.cc:2229] T cfff7f3301b3420db337ede054013b10 P 3ba34efe46904a9581b54eea2f6b6cda [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:20:20.839107 13706 raft_consensus.cc:2258] T cfff7f3301b3420db337ede054013b10 P 3ba34efe46904a9581b54eea2f6b6cda [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:20:20.840309 13707 ts_tablet_manager.cc:1905] T cfff7f3301b3420db337ede054013b10 P 4e4d9d282a65497e9189190a01a26bab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:20:20.840351 13706 ts_tablet_manager.cc:1905] T cfff7f3301b3420db337ede054013b10 P 3ba34efe46904a9581b54eea2f6b6cda: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:20:20.843022 13707 ts_tablet_manager.cc:1918] T cfff7f3301b3420db337ede054013b10 P 4e4d9d282a65497e9189190a01a26bab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:20:20.843021 13706 ts_tablet_manager.cc:1918] T cfff7f3301b3420db337ede054013b10 P 3ba34efe46904a9581b54eea2f6b6cda: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:20:20.843120 13707 log.cc:1196] T cfff7f3301b3420db337ede054013b10 P 4e4d9d282a65497e9189190a01a26bab: Deleting WAL directory at /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-1/wal/wals/cfff7f3301b3420db337ede054013b10
I20260618 22:20:20.843120 13706 log.cc:1196] T cfff7f3301b3420db337ede054013b10 P 3ba34efe46904a9581b54eea2f6b6cda: Deleting WAL directory at /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781821065380026-393-0/minicluster-data/ts-2/wal/wals/cfff7f3301b3420db337ede054013b10
I20260618 22:20:20.843444 13707 ts_tablet_manager.cc:1939] T cfff7f3301b3420db337ede054013b10 P 4e4d9d282a65497e9189190a01a26bab: Deleting consensus metadata
I20260618 22:20:20.843444 13706 ts_tablet_manager.cc:1939] T cfff7f3301b3420db337ede054013b10 P 3ba34efe46904a9581b54eea2f6b6cda: Deleting consensus metadata
W20260618 22:20:20.843784 13158 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.65:40163 (error 108)
I20260618 22:20:20.844897 13158 catalog_manager.cc:4841] TS 3ba34efe46904a9581b54eea2f6b6cda (127.0.98.67:35107): tablet cfff7f3301b3420db337ede054013b10 (table test-table [id=db8f4b9b9c034c3bae2630b4b6b77f24]) successfully deleted
I20260618 22:20:20.845088 13158 catalog_manager.cc:4841] TS 4e4d9d282a65497e9189190a01a26bab (127.0.98.66:34817): tablet cfff7f3301b3420db337ede054013b10 (table test-table [id=db8f4b9b9c034c3bae2630b4b6b77f24]) successfully deleted
I20260618 22:20:20.845571 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 13374
I20260618 22:20:20.845610 13671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:20.845333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:40163 (local address 127.0.0.1:60860)
0618 22:20:20.845461 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:20.845530 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.65:40163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260618 22:20:20.847388 13158 catalog_manager.cc:4584] TS 7d0f3194f19141c697b072ecc2faea28 (127.0.98.65:40163): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cfff7f3301b3420db337ede054013b10: Network error: Client connection negotiation failed: client connection to 127.0.98.65:40163: connect: Connection refused (error 111)
I20260618 22:20:20.851402 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 13509
I20260618 22:20:20.858098 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 13141
I20260618 22:20:20.864341 393 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260618 22:20:21.199258 393 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260618 22:20:21.199366 393 mini_ranger.cc:66] Stopping Ranger...
I20260618 22:20:21.545572 393 mini_ranger.cc:68] Stopped Ranger
2026-06-18 22:20:21.545 UTC [12208] LOG: received smart shutdown request
2026-06-18 22:20:21.549 UTC [12208] LOG: background worker "logical replication launcher" (PID 12217) exited with exit code 1
2026-06-18 22:20:21.550 UTC [12212] LOG: shutting down
2026-06-18 22:20:21.706 UTC [12208] LOG: database system is shut down
2026-06-18T22:20:21Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (47879 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821065380026-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:20:21 dist-test-slave-g6r4 krb5kdc[13718](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:20:21 dist-test-slave-g6r4 krb5kdc[13718](info): set up 2 sockets
Jun 18 22:20:21 dist-test-slave-g6r4 krb5kdc[13718](info): commencing operation
krb5kdc: starting...
W20260618 22:20:22.862001 393 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821065380026-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821065380026-393-0/krb5kdc/test-user.keytab.
Jun 18 22:20:22 dist-test-slave-g6r4 krb5kdc[13718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:20:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:20:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260618 22:20:22.986930 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821065380026-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821065380026-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821065380026-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:40419
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:43847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:40419,127.0.98.125:43265,127.0.98.124:40849
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:20:23.093111 13731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:20:23.093390 13731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:20:23.093470 13731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:20:23.097469 13731 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:20:23.097556 13731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:20:23.097591 13731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:20:23.097630 13731 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:20:23.097666 13731 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:20:23.102753 13731 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:43847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821065380026-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821065380026-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:40419,127.0.98.125:43265,127.0.98.124:40849
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:40419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821065380026-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:20:23.103753 13731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:20:23.104768 13731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:20:23.112836 13736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:20:23.112846 13739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:20:23.112929 13737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:20:23.113220 13731 server_base.cc:1034] running on GCE node
I20260618 22:20:23.113668 13731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:20:23.114715 13731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:20:23.115870 13731 hybrid_clock.cc:648] HybridClock initialized: now 1781821223115852 us; error 31 us; skew 500 ppm
Jun 18 22:20:23 dist-test-slave-g6r4 krb5kdc[13718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821223, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:20:23.118487 13731 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260618 22:20:23.119283 13731 webserver.cc:460] Webserver started at http://127.0.98.126:42763/ using document root <none> and password file <none>
I20260618 22:20:23.119804 13731 fs_manager.cc:359] Metadata directory not provided
I20260618 22:20:23.119890 13731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:20:23.120105 13731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:20:23.122104 13731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821065380026-393-0/minicluster-data/master-0/data/instance:
uuid: "518d01444eef4511b90ee405aac52735"
format_stamp: "Formatted at 2026-06-18 22:20:23 on dist-test-slave-g6r4"
server_key: "ca4541d8cf1d025674963b663ab8d62c"
server_key_iv: "1b5e821a928935fdae096efdd233aa13"
server_key_version: "encryptionkey@0"
I20260618 22:20:23.122643 13731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821065380026-393-0/minicluster-data/master-0/wal/instance:
uuid: "518d01444eef4511b90ee405aac52735"
format_stamp: "Formatted at 2026-06-18 22:20:23 on dist-test-slave-g6r4"
server_key: "ca4541d8cf1d025674963b663ab8d62c"
server_key_iv: "1b5e821a928935fdae096efdd233aa13"
server_key_version: "encryptionkey@0"
I20260618 22:20:23.126135 13731 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:20:23.128309 13746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:20:23.129390 13731 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:20:23.129518 13731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821065380026-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821065380026-393-0/minicluster-data/master-0/wal
uuid: "518d01444eef4511b90ee405aac52735"
format_stamp: "Formatted at 2026-06-18 22:20:23 on dist-test-slave-g6r4"
server_key: "ca4541d8cf1d025674963b663ab8d62c"
server_key_iv: "1b5e821a928935fdae096efdd233aa13"
server_key_version: "encryptionkey@0"
I20260618 22:20:23.129619 13731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821065380026-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821065380026-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821065380026-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:20:23.146248 13731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:20:23.148736 13731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:20:23.148924 13731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:20:23.155647 13731 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:40419
I20260618 22:20:23.155674 13798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:40419 every 8 connection(s)
I20260618 22:20:23.156530 13731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
I20260618 22:20:23.159046 13799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:20:23.162258 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 13731
I20260618 22:20:23.162393 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821065380026-393-0/minicluster-data/master-0/wal/instance
I20260618 22:20:23.162662 393 external_mini_cluster.cc:1428] Setting key e06f6bf2e537287c5ebc114c1092fc06
I20260618 22:20:23.162598 13799 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40419 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.125@KRBTEST.COM" created.
Jun 18 22:20:23 dist-test-slave-g6r4 krb5kdc[13718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821223, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:20:23.179651 13803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:23.164766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43704 (local address 127.0.98.126:40419)
0618 22:20:23.165266 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0618 22:20:23.165282 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0618 22:20:23.165419 (+ 137us) server_negotiation.cc:407] Connection header received
0618 22:20:23.166822 (+ 1403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:20:23.166826 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:20:23.166991 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:20:23.167136 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:20:23.168248 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:23.169086 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:23.170584 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:23.170987 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:23.171112 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:23.174565 (+ 3453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:20:23.174570 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:20:23.174579 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:20:23.174598 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:20:23.176437 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:23.176987 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:23.176996 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:23.177001 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:23.177140 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:23.177581 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:23.177584 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:23.177586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:23.178116 (+ 530us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:20:23.178277 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0618 22:20:23.178697 (+ 420us) server_negotiation.cc:299] Negotiation successful
0618 22:20:23.178964 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":377,"spinlock_wait_cycles":48512,"thread_start_us":147,"threads_started":1}
I20260618 22:20:23.179651 13802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:23.164766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:40419 (local address 127.0.0.1:43704)
0618 22:20:23.165261 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:23.165312 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0618 22:20:23.166513 (+ 1201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:20:23.167196 (+ 683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:20:23.167205 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:20:23.167695 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:20:23.168071 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:23.168093 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:23.169222 (+ 1129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:23.169226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:23.170476 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:23.170486 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:23.171148 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:23.171151 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:23.171259 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:23.172004 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:20:23.172024 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:20:23.174378 (+ 2354us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:20:23.176582 (+ 2204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:20:23.176593 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:20:23.176601 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:20:23.176853 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:20:23.177271 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:20:23.177276 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:20:23.177279 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:20:23.177448 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:20:23.178259 (+ 811us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:20:23.178265 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:20:23.178409 (+ 144us) client_negotiation.cc:769] Sending connection context
0618 22:20:23.178570 (+ 161us) client_negotiation.cc:240] Negotiation successful
0618 22:20:23.178839 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":133,"threads_started":1}
I20260618 22:20:23.182273 13799 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.125" port: 43265 } has no permanent_uuid. Determining permanent_uuid...
I20260618 22:20:23.182996 13802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:23.182760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:43265 (local address 127.0.0.1:56968)
0618 22:20:23.182882 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:23.182944 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.125:43265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260618 22:20:23.183185 13749 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.125:43265: connect: Connection refused (error 111)
I20260618 22:20:23.184063 13802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:23.183849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:43265 (local address 127.0.0.1:56982)
0618 22:20:23.183963 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:23.184017 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.125:43265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260618 22:20:23.184293 13799 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.125:43265: Network error: Client connection negotiation failed: client connection to 127.0.98.125:43265: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821065380026-393-0/krb5kdc/kudu_127.0.98.125.keytab.
Entry for principal kudu/127.0.98.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821065380026-393-0/krb5kdc/kudu_127.0.98.125.keytab.
I20260618 22:20:23.188911 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821065380026-393-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821065380026-393-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821065380026-393-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821065380026-393-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.125:43265
--webserver_interface=127.0.98.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821065380026-393-0/krb5kdc/kudu_127.0.98.125.keytab
--principal=kudu/127.0.98.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:43847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:40419,127.0.98.125:43265,127.0.98.124:40849
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 22:20:23.233552 13799 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.125" port: 43265 } attempt: 1
I20260618 22:20:23.234517 13802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:23.234157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:43265 (local address 127.0.0.1:56992)
0618 22:20:23.234352 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:23.234425 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.125:43265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260618 22:20:23.235282 13802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:23.235052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:43265 (local address 127.0.0.1:56996)
0618 22:20:23.235166 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:23.235220 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.125:43265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260618 22:20:23.235450 13799 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.125:43265: Network error: Client connection negotiation failed: client connection to 127.0.98.125:43265: connect: Connection refused (error 111)
W20260618 22:20:23.292606 13806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:20:23.292846 13806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:20:23.292883 13806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:20:23.296767 13806 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:20:23.296813 13806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:20:23.296833 13806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:20:23.296888 13806 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:20:23.296924 13806 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:20:23.301980 13806 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:43847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821065380026-393-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821065380026-393-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:40419,127.0.98.125:43265,127.0.98.124:40849
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821065380026-393-0/krb5kdc/kudu_127.0.98.125.keytab
--principal=kudu/127.0.98.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.125:43265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821065380026-393-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13806
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821065380026-393-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:20:23.302917 13806 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:20:23.303642 13799 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.125" port: 43265 } attempt: 2
I20260618 22:20:23.303931 13806 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260618 22:20:23.304471 13802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:23.304199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:43265 (local address 127.0.0.1:57012)
0618 22:20:23.304341 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:23.304401 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.125:43265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260618 22:20:23.305173 13802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:23.304997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:43265 (local address 127.0.0.1:57026)
0618 22:20:23.305085 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:23.305126 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.125:43265: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260618 22:20:23.305315 13799 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.125:43265: Network error: Client connection negotiation failed: client connection to 127.0.98.125:43265: connect: Connection refused (error 111)
W20260618 22:20:23.312316 13812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:20:23.312304 13811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:20:23.312367 13814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:20:23.312610 13806 server_base.cc:1034] running on GCE node
I20260618 22:20:23.313102 13806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:20:23.314260 13806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:20:23.315419 13806 hybrid_clock.cc:648] HybridClock initialized: now 1781821223315407 us; error 22 us; skew 500 ppm
Jun 18 22:20:23 dist-test-slave-g6r4 krb5kdc[13718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821223, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:20:23.317984 13806 init.cc:377] Logged in from keytab as kudu/127.0.98.125@KRBTEST.COM (short username kudu)
I20260618 22:20:23.318781 13806 webserver.cc:460] Webserver started at http://127.0.98.125:34895/ using document root <none> and password file <none>
I20260618 22:20:23.319309 13806 fs_manager.cc:359] Metadata directory not provided
I20260618 22:20:23.319399 13806 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:20:23.319620 13806 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:20:23.321483 13806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821065380026-393-0/minicluster-data/master-1/data/instance:
uuid: "ba7dbd10493740cbaa0190c0f67fb10a"
format_stamp: "Formatted at 2026-06-18 22:20:23 on dist-test-slave-g6r4"
server_key: "967089b6ebd520410bd7b75138471157"
server_key_iv: "39be6370b1b1bd5ae01d5a87d3f1de71"
server_key_version: "encryptionkey@0"
I20260618 22:20:23.322100 13806 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821065380026-393-0/minicluster-data/master-1/wal/instance:
uuid: "ba7dbd10493740cbaa0190c0f67fb10a"
format_stamp: "Formatted at 2026-06-18 22:20:23 on dist-test-slave-g6r4"
server_key: "967089b6ebd520410bd7b75138471157"
server_key_iv: "39be6370b1b1bd5ae01d5a87d3f1de71"
server_key_version: "encryptionkey@0"
I20260618 22:20:23.325695 13806 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:20:23.328284 13821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:20:23.329460 13806 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:20:23.329604 13806 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821065380026-393-0/minicluster-data/master-1/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821065380026-393-0/minicluster-data/master-1/wal
uuid: "ba7dbd10493740cbaa0190c0f67fb10a"
format_stamp: "Formatted at 2026-06-18 22:20:23 on dist-test-slave-g6r4"
server_key: "967089b6ebd520410bd7b75138471157"
server_key_iv: "39be6370b1b1bd5ae01d5a87d3f1de71"
server_key_version: "encryptionkey@0"
I20260618 22:20:23.329727 13806 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821065380026-393-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821065380026-393-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821065380026-393-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:20:23.358403 13806 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:20:23.361123 13806 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:20:23.361330 13806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:20:23.368180 13806 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.125:43265
I20260618 22:20:23.368207 13873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.125:43265 every 8 connection(s)
I20260618 22:20:23.369066 13806 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821065380026-393-0/minicluster-data/master-1/data/info.pb
I20260618 22:20:23.371595 13874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:20:23.374437 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 13806
I20260618 22:20:23.374575 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821065380026-393-0/minicluster-data/master-1/wal/instance
I20260618 22:20:23.374861 393 external_mini_cluster.cc:1428] Setting key bc5aa39cc1ff0a6b21fd9d7b126d3b7d
I20260618 22:20:23.375257 13874 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40419 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.124@KRBTEST.COM" created.
Jun 18 22:20:23 dist-test-slave-g6r4 krb5kdc[13718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821223, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.125@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:20:23.390043 13803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:23.377063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43714 (local address 127.0.98.126:40419)
0618 22:20:23.377214 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0618 22:20:23.377218 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:20:23.378040 (+ 822us) server_negotiation.cc:407] Connection header received
0618 22:20:23.378909 (+ 869us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:20:23.378914 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:20:23.378977 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:20:23.379082 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:20:23.380108 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:23.380985 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:23.382115 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:23.382496 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:23.382626 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:23.386341 (+ 3715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:20:23.386345 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:20:23.386347 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:20:23.386371 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:20:23.387809 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:23.388329 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:23.388336 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:23.388341 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:23.388438 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:23.388765 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:23.388772 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:23.388776 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:23.389031 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:20:23.389146 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0618 22:20:23.389614 (+ 468us) server_negotiation.cc:299] Negotiation successful
0618 22:20:23.389756 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260618 22:20:23.390576 13877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:23.377439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:40419 (local address 127.0.0.1:43714)
0618 22:20:23.377864 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:23.377936 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0618 22:20:23.378714 (+ 778us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:20:23.379213 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:20:23.379218 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:20:23.379521 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:20:23.379960 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:23.379972 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:23.381103 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:23.381105 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:23.381998 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:23.382007 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:23.382686 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:23.382690 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:23.382827 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:23.383338 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:20:23.383354 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:20:23.386170 (+ 2816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:20:23.387946 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:20:23.387959 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:20:23.387971 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:20:23.388212 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:20:23.388527 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:20:23.388529 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:20:23.388530 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:20:23.388647 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:20:23.389131 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:20:23.389136 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:20:23.389353 (+ 217us) client_negotiation.cc:769] Sending connection context
0618 22:20:23.389491 (+ 138us) client_negotiation.cc:240] Negotiation successful
0618 22:20:23.389726 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":126,"threads_started":1}
I20260618 22:20:23.391893 13874 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.125" port: 43265 } has no permanent_uuid. Determining permanent_uuid...
I20260618 22:20:23.396478 13799 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.125" port: 43265 } attempt: 3
Jun 18 22:20:23 dist-test-slave-g6r4 krb5kdc[13718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821223, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.125@KRBTEST.COM for kudu/127.0.98.125@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821065380026-393-0/krb5kdc/kudu_127.0.98.124.keytab.
Entry for principal kudu/127.0.98.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821065380026-393-0/krb5kdc/kudu_127.0.98.124.keytab.
I20260618 22:20:23.401996 13877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:23.392360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:43265 (local address 127.0.0.1:57032)
0618 22:20:23.392768 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:23.392785 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0618 22:20:23.392864 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:20:23.393210 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:20:23.393212 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:20:23.393371 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:20:23.393453 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:23.393458 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:23.394551 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:23.394558 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:23.395092 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:23.395099 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:23.395643 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:23.395646 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:23.395749 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:23.396222 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:20:23.396232 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:20:23.397589 (+ 1357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:20:23.399839 (+ 2250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:20:23.399843 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:20:23.399844 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:20:23.400185 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:20:23.400511 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:20:23.400514 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:20:23.400516 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:20:23.400621 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:20:23.401205 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:20:23.401208 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:20:23.401289 (+ 81us) client_negotiation.cc:769] Sending connection context
0618 22:20:23.401662 (+ 373us) client_negotiation.cc:240] Negotiation successful
0618 22:20:23.401801 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260618 22:20:23.402546 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821065380026-393-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821065380026-393-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821065380026-393-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821065380026-393-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.124:40849
--webserver_interface=127.0.98.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821065380026-393-0/krb5kdc/kudu_127.0.98.124.keytab
--principal=kudu/127.0.98.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:43847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:40419,127.0.98.125:43265,127.0.98.124:40849
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 22:20:23.402773 13879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:23.392486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57032 (local address 127.0.98.125:43265)
0618 22:20:23.392763 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0618 22:20:23.392775 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0618 22:20:23.392834 (+ 59us) server_negotiation.cc:407] Connection header received
0618 22:20:23.392937 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:20:23.392942 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:20:23.393088 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:20:23.393223 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:20:23.393556 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:23.394442 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:23.395206 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:23.395537 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:23.395647 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:23.397694 (+ 2047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:20:23.397702 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:20:23.397714 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:20:23.397740 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:20:23.399712 (+ 1972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:23.400307 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:23.400313 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:23.400315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:23.400402 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:23.400726 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:23.400728 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:23.400730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:23.401013 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:20:23.401241 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0618 22:20:23.402068 (+ 827us) server_negotiation.cc:299] Negotiation successful
0618 22:20:23.402240 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":122,"threads_started":1}
Jun 18 22:20:23 dist-test-slave-g6r4 krb5kdc[13718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821223, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for kudu/127.0.98.125@KRBTEST.COM
I20260618 22:20:23.404860 13874 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40849 } has no permanent_uuid. Determining permanent_uuid...
I20260618 22:20:23.405500 13877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:23.405269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:40849 (local address 127.0.0.1:55612)
0618 22:20:23.405377 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:23.405451 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:40849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260618 22:20:23.405717 13823 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.124:40849: connect: Connection refused (error 111)
I20260618 22:20:23.406502 13877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:23.406304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:40849 (local address 127.0.0.1:55618)
0618 22:20:23.406405 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:23.406448 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:40849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260618 22:20:23.407029 13802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:23.396950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:43265 (local address 127.0.0.1:57038)
0618 22:20:23.397088 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:23.397107 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0618 22:20:23.397210 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:20:23.397551 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:20:23.397554 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:20:23.397682 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:20:23.398326 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:23.398346 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:23.399539 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:23.399544 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:23.400430 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:23.400442 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:23.401095 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:23.401100 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:23.401229 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:23.401656 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:20:23.401676 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:20:23.403278 (+ 1602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:20:23.405387 (+ 2109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:20:23.405393 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:20:23.405397 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:20:23.405657 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:20:23.406006 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:20:23.406012 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:20:23.406017 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:20:23.406110 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:20:23.406587 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:20:23.406590 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:20:23.406699 (+ 109us) client_negotiation.cc:769] Sending connection context
0618 22:20:23.406821 (+ 122us) client_negotiation.cc:240] Negotiation successful
0618 22:20:23.406922 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260618 22:20:23.407284 13880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:23.397002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57038 (local address 127.0.98.125:43265)
0618 22:20:23.397319 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0618 22:20:23.397330 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0618 22:20:23.397345 (+ 15us) server_negotiation.cc:407] Connection header received
0618 22:20:23.397377 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:20:23.397380 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:20:23.397420 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:20:23.397509 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:20:23.398467 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:23.399394 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:23.400548 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:23.400966 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:23.401068 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:23.403428 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:20:23.403432 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:20:23.403435 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:20:23.403457 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:20:23.405256 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:23.405786 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:23.405792 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:23.405796 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:23.405879 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:23.406234 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:23.406240 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:23.406244 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:23.406476 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:20:23.406582 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0618 22:20:23.406904 (+ 322us) server_negotiation.cc:299] Negotiation successful
0618 22:20:23.407035 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":104,"threads_started":1}
W20260618 22:20:23.407318 13874 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.124:40849: Network error: Client connection negotiation failed: client connection to 127.0.98.124:40849: connect: Connection refused (error 111)
I20260618 22:20:23.407927 13799 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40849 } has no permanent_uuid. Determining permanent_uuid...
I20260618 22:20:23.408587 13802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:23.408386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:40849 (local address 127.0.0.1:55632)
0618 22:20:23.408492 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:23.408535 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:40849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260618 22:20:23.409276 13802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:23.409049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:40849 (local address 127.0.0.1:55648)
0618 22:20:23.409185 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:23.409233 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:40849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260618 22:20:23.409411 13799 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.124:40849: Network error: Client connection negotiation failed: client connection to 127.0.98.124:40849: connect: Connection refused (error 111)
I20260618 22:20:23.440554 13799 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40849 } attempt: 1
I20260618 22:20:23.441396 13802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:23.441068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:40849 (local address 127.0.0.1:55652)
0618 22:20:23.441264 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:23.441324 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:40849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260618 22:20:23.442126 13802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:23.441874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:40849 (local address 127.0.0.1:55668)
0618 22:20:23.442030 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:23.442077 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:40849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20260618 22:20:23.442256 13799 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.124:40849: Network error: Client connection negotiation failed: client connection to 127.0.98.124:40849: connect: Connection refused (error 111)
I20260618 22:20:23.456668 13874 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40849 } attempt: 1
I20260618 22:20:23.457439 13877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:23.457134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:40849 (local address 127.0.0.1:55682)
0618 22:20:23.457290 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:23.457357 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:40849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260618 22:20:23.458177 13877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:23.457954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:40849 (local address 127.0.0.1:55690)
0618 22:20:23.458063 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:23.458116 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:40849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260618 22:20:23.458341 13874 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.124:40849: Network error: Client connection negotiation failed: client connection to 127.0.98.124:40849: connect: Connection refused (error 111)
W20260618 22:20:23.511284 13881 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:20:23.511505 13881 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:20:23.511579 13881 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:20:23.515451 13881 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:20:23.515498 13881 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:20:23.515544 13881 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:20:23.515580 13881 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:20:23.515612 13881 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:20:23.517400 13799 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40849 } attempt: 2
I20260618 22:20:23.518182 13802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:23.517852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:40849 (local address 127.0.0.1:55694)
0618 22:20:23.518006 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:23.518075 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:40849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260618 22:20:23.518898 13802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:23.518707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:40849 (local address 127.0.0.1:55700)
0618 22:20:23.518797 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:23.518854 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:40849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260618 22:20:23.519038 13799 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.124:40849: Network error: Client connection negotiation failed: client connection to 127.0.98.124:40849: connect: Connection refused (error 111)
I20260618 22:20:23.521078 13881 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:43847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821065380026-393-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821065380026-393-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:40419,127.0.98.125:43265,127.0.98.124:40849
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821065380026-393-0/krb5kdc/kudu_127.0.98.124.keytab
--principal=kudu/127.0.98.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.124:40849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821065380026-393-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13881
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821065380026-393-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:20:23.522038 13881 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:20:23.522962 13881 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260618 22:20:23.526495 13874 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40849 } attempt: 2
I20260618 22:20:23.527228 13877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:23.526907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:40849 (local address 127.0.0.1:55710)
0618 22:20:23.527057 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:23.527126 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:40849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260618 22:20:23.528013 13877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:23.527779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:40849 (local address 127.0.0.1:55720)
0618 22:20:23.527892 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:23.527945 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:40849: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260618 22:20:23.528196 13874 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.124:40849: Network error: Client connection negotiation failed: client connection to 127.0.98.124:40849: connect: Connection refused (error 111)
I20260618 22:20:23.531571 13881 server_base.cc:1034] running on GCE node
W20260618 22:20:23.531491 13888 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:20:23.531529 13887 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:20:23.531492 13890 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:20:23.532279 13881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:20:23.533425 13881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:20:23.534601 13881 hybrid_clock.cc:648] HybridClock initialized: now 1781821223534564 us; error 47 us; skew 500 ppm
Jun 18 22:20:23 dist-test-slave-g6r4 krb5kdc[13718](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821223, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:20:23.537223 13881 init.cc:377] Logged in from keytab as kudu/127.0.98.124@KRBTEST.COM (short username kudu)
I20260618 22:20:23.538075 13881 webserver.cc:460] Webserver started at http://127.0.98.124:33333/ using document root <none> and password file <none>
I20260618 22:20:23.538628 13881 fs_manager.cc:359] Metadata directory not provided
I20260618 22:20:23.538718 13881 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:20:23.538930 13881 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:20:23.540753 13881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821065380026-393-0/minicluster-data/master-2/data/instance:
uuid: "5caa1743702b4718b3225cb48dcf31f6"
format_stamp: "Formatted at 2026-06-18 22:20:23 on dist-test-slave-g6r4"
server_key: "f1fe1714f06a060c957f24a060d634ec"
server_key_iv: "ff2c25a003460479799e73ed370b5e7f"
server_key_version: "encryptionkey@0"
I20260618 22:20:23.541287 13881 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821065380026-393-0/minicluster-data/master-2/wal/instance:
uuid: "5caa1743702b4718b3225cb48dcf31f6"
format_stamp: "Formatted at 2026-06-18 22:20:23 on dist-test-slave-g6r4"
server_key: "f1fe1714f06a060c957f24a060d634ec"
server_key_iv: "ff2c25a003460479799e73ed370b5e7f"
server_key_version: "encryptionkey@0"
I20260618 22:20:23.544638 13881 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260618 22:20:23.546875 13897 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:20:23.548022 13881 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:20:23.548166 13881 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821065380026-393-0/minicluster-data/master-2/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821065380026-393-0/minicluster-data/master-2/wal
uuid: "5caa1743702b4718b3225cb48dcf31f6"
format_stamp: "Formatted at 2026-06-18 22:20:23 on dist-test-slave-g6r4"
server_key: "f1fe1714f06a060c957f24a060d634ec"
server_key_iv: "ff2c25a003460479799e73ed370b5e7f"
server_key_version: "encryptionkey@0"
I20260618 22:20:23.548270 13881 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821065380026-393-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821065380026-393-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821065380026-393-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:20:23.575591 13881 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:20:23.578831 13881 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:20:23.579030 13881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:20:23.585753 13881 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.124:40849
I20260618 22:20:23.585776 13949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.124:40849 every 8 connection(s)
I20260618 22:20:23.586670 13881 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821065380026-393-0/minicluster-data/master-2/data/info.pb
I20260618 22:20:23.589215 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 13881
I20260618 22:20:23.589236 13950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:20:23.589345 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781821065380026-393-0/minicluster-data/master-2/wal/instance
I20260618 22:20:23.589619 393 external_mini_cluster.cc:1428] Setting key dbd43d3eda402c26bf550e8a4afc1ec6
I20260618 22:20:23.594275 13950 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40419 } has no permanent_uuid. Determining permanent_uuid...
Jun 18 22:20:23 dist-test-slave-g6r4 krb5kdc[13718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.124@KRBTEST.COM
Jun 18 22:20:23 dist-test-slave-g6r4 krb5kdc[13718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821223, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.124@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:20:23.604271 13953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:23.590959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55734 (local address 127.0.98.124:40849)
0618 22:20:23.591374 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0618 22:20:23.591395 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0618 22:20:23.591436 (+ 41us) server_negotiation.cc:407] Connection header received
0618 22:20:23.592114 (+ 678us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:20:23.592136 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:20:23.592555 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:20:23.592904 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:20:23.593521 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:23.595012 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:23.595941 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:23.596559 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:23.596772 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:23.599366 (+ 2594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:20:23.599370 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:20:23.599379 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:20:23.599402 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:20:23.601203 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:23.601641 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:23.601649 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:23.601658 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:23.601776 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:23.602086 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:23.602089 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:23.602091 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:23.602436 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:20:23.602665 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0618 22:20:23.602993 (+ 328us) server_negotiation.cc:299] Negotiation successful
0618 22:20:23.603275 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":107,"threads_started":1}
I20260618 22:20:23.605619 13803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:23.596089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55438 (local address 127.0.98.126:40419)
0618 22:20:23.596228 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0618 22:20:23.596232 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:20:23.596501 (+ 269us) server_negotiation.cc:407] Connection header received
0618 22:20:23.596616 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:20:23.596622 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:20:23.596697 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:20:23.596772 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:20:23.597573 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:23.598119 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:23.599058 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:23.599395 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:23.599472 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:23.601729 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:20:23.601732 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:20:23.601734 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:20:23.601759 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:20:23.603893 (+ 2134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:23.604362 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:23.604366 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:23.604368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:23.604458 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:23.604765 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:23.604769 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:23.604771 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:23.605008 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:20:23.605106 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0618 22:20:23.605422 (+ 316us) server_negotiation.cc:299] Negotiation successful
0618 22:20:23.605515 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260618 22:20:23.605659 13954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:23.596018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:40419 (local address 127.0.0.1:55438)
0618 22:20:23.596378 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:23.596423 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0618 22:20:23.596534 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:20:23.596833 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:20:23.596839 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:20:23.597267 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:20:23.597440 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:23.597451 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:23.598241 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:23.598247 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:23.598941 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:23.598950 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:23.599492 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:23.599496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:23.599624 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:23.600119 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:20:23.600137 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:20:23.601594 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:20:23.604009 (+ 2415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:20:23.604023 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:20:23.604028 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:20:23.604258 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:20:23.604570 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:20:23.604573 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:20:23.604575 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:20:23.604661 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:20:23.605108 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:20:23.605112 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:20:23.605213 (+ 101us) client_negotiation.cc:769] Sending connection context
0618 22:20:23.605350 (+ 137us) client_negotiation.cc:240] Negotiation successful
0618 22:20:23.605508 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":152,"threads_started":1}
I20260618 22:20:23.606649 13950 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.125" port: 43265 } has no permanent_uuid. Determining permanent_uuid...
Jun 18 22:20:23 dist-test-slave-g6r4 krb5kdc[13718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821223, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.124@KRBTEST.COM for kudu/127.0.98.125@KRBTEST.COM
Jun 18 22:20:23 dist-test-slave-g6r4 krb5kdc[13718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:20:23.614784 13954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:23.607159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:43265 (local address 127.0.0.1:42748)
0618 22:20:23.607314 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:23.607329 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:20:23.607398 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:20:23.607618 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:20:23.607621 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:20:23.607797 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:20:23.607876 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:23.607881 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:23.608615 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:23.608622 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:23.609132 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:23.609139 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:23.609624 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:23.609628 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:23.609740 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:23.610111 (+ 371us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:20:23.610121 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:20:23.611300 (+ 1179us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:20:23.613175 (+ 1875us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:20:23.613199 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:20:23.613201 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:20:23.613450 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:20:23.613723 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:20:23.613726 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:20:23.613729 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:20:23.613815 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:20:23.614289 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:20:23.614295 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:20:23.614389 (+ 94us) client_negotiation.cc:769] Sending connection context
0618 22:20:23.614491 (+ 102us) client_negotiation.cc:240] Negotiation successful
0618 22:20:23.614612 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260618 22:20:23.614809 13880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:23.607266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42748 (local address 127.0.98.125:43265)
0618 22:20:23.607401 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0618 22:20:23.607405 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:20:23.607421 (+ 16us) server_negotiation.cc:407] Connection header received
0618 22:20:23.607469 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:20:23.607472 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:20:23.607513 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:20:23.607605 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:20:23.607993 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:23.608508 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:23.609257 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:23.609515 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:23.609625 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:23.611432 (+ 1807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:20:23.611435 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:20:23.611437 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:20:23.611459 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:20:23.613063 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:23.613552 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:23.613555 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:23.613556 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:23.613626 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:23.613940 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:23.613946 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:23.613953 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:23.614176 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:20:23.614283 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0618 22:20:23.614550 (+ 267us) server_negotiation.cc:299] Negotiation successful
0618 22:20:23.614696 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260618 22:20:23.615525 13950 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40849 } has no permanent_uuid. Determining permanent_uuid...
I20260618 22:20:23.615764 13803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:23.606709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55446 (local address 127.0.98.126:40419)
0618 22:20:23.606805 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0618 22:20:23.606811 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:20:23.606844 (+ 33us) server_negotiation.cc:407] Connection header received
0618 22:20:23.606971 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:20:23.606974 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:20:23.607011 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:20:23.607096 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:20:23.607767 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:23.608243 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:23.609012 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:23.609242 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:23.609348 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:23.612034 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:20:23.612038 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:20:23.612041 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:20:23.612064 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:20:23.613816 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:23.614321 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:23.614324 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:23.614325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:23.614409 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:23.614704 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:23.614712 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:23.614714 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:23.614934 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:20:23.615157 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0618 22:20:23.615515 (+ 358us) server_negotiation.cc:299] Negotiation successful
0618 22:20:23.615626 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260618 22:20:23.618185 13799 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40849 } attempt: 3
I20260618 22:20:23.619372 13874 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40849 } attempt: 3
Jun 18 22:20:23 dist-test-slave-g6r4 krb5kdc[13718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821223, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.124@KRBTEST.COM for kudu/127.0.98.124@KRBTEST.COM
Jun 18 22:20:23 dist-test-slave-g6r4 krb5kdc[13718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.125@KRBTEST.COM
Jun 18 22:20:23 dist-test-slave-g6r4 krb5kdc[13718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821223, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for kudu/127.0.98.124@KRBTEST.COM
Jun 18 22:20:23 dist-test-slave-g6r4 krb5kdc[13718](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821223, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.125@KRBTEST.COM for kudu/127.0.98.124@KRBTEST.COM
I20260618 22:20:23.626212 13953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:23.616071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55742 (local address 127.0.98.124:40849)
0618 22:20:23.616189 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0618 22:20:23.616200 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0618 22:20:23.616500 (+ 300us) server_negotiation.cc:407] Connection header received
0618 22:20:23.616600 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:20:23.616603 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:20:23.616635 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:20:23.616728 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:20:23.617096 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:23.617980 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:23.619160 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:23.619505 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:23.619603 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:23.622533 (+ 2930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:20:23.622537 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:20:23.622539 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:20:23.622558 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:20:23.624491 (+ 1933us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:23.624921 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:23.624924 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:23.624927 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:23.625000 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:23.625298 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:23.625301 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:23.625303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:23.625503 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:20:23.625577 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0618 22:20:23.625935 (+ 358us) server_negotiation.cc:299] Negotiation successful
0618 22:20:23.626043 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":40}
I20260618 22:20:23.626724 13954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:23.616002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:40849 (local address 127.0.0.1:55742)
0618 22:20:23.616444 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:23.616459 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:20:23.616533 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:20:23.616747 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:20:23.616750 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:20:23.616903 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:20:23.616981 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:23.616988 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:23.618091 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:23.618097 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:23.619039 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:23.619048 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:23.619677 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:23.619681 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:23.619754 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:23.620442 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:20:23.620461 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:20:23.622423 (+ 1962us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:20:23.624599 (+ 2176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:20:23.624603 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:20:23.624606 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:20:23.624823 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:20:23.625107 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:20:23.625110 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:20:23.625112 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:20:23.625208 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:20:23.625638 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:20:23.625640 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:20:23.625721 (+ 81us) client_negotiation.cc:769] Sending connection context
0618 22:20:23.626387 (+ 666us) client_negotiation.cc:240] Negotiation successful
0618 22:20:23.626541 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371}
I20260618 22:20:23.628410 13880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:23.616904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42750 (local address 127.0.98.125:43265)
0618 22:20:23.617024 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0618 22:20:23.617028 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:20:23.617041 (+ 13us) server_negotiation.cc:407] Connection header received
0618 22:20:23.617122 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:20:23.617125 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:20:23.617159 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:20:23.617247 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:20:23.617803 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:23.618489 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:23.619252 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:23.619572 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:23.619667 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:23.623023 (+ 3356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:20:23.623027 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:20:23.623029 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:20:23.623052 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:20:23.625655 (+ 2603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:23.626140 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:23.626144 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:23.626146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:23.626227 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:23.626680 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:23.626684 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:23.626686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:23.627299 (+ 613us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:20:23.627494 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0618 22:20:23.628123 (+ 629us) server_negotiation.cc:299] Negotiation successful
0618 22:20:23.628241 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260618 22:20:23.629724 13877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:23.619725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:40849 (local address 127.0.0.1:55762)
0618 22:20:23.619895 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:23.619908 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:20:23.619974 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:20:23.620749 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:20:23.620752 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:20:23.620923 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:20:23.621005 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:23.621011 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:23.622084 (+ 1073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:23.622090 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:23.622905 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:23.622918 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:23.623420 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:23.623425 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:23.623514 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:23.624047 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:20:23.624061 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:20:23.625295 (+ 1234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:20:23.628283 (+ 2988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:20:23.628287 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:20:23.628289 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:20:23.628509 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:20:23.628802 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:20:23.628806 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:20:23.628808 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:20:23.628890 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:20:23.629281 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:20:23.629284 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:20:23.629365 (+ 81us) client_negotiation.cc:769] Sending connection context
0618 22:20:23.629445 (+ 80us) client_negotiation.cc:240] Negotiation successful
0618 22:20:23.629571 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112}
I20260618 22:20:23.630220 13950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5caa1743702b4718b3225cb48dcf31f6: Bootstrap starting.
I20260618 22:20:23.630291 13956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:23.620166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55762 (local address 127.0.98.124:40849)
0618 22:20:23.620511 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0618 22:20:23.620521 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0618 22:20:23.620535 (+ 14us) server_negotiation.cc:407] Connection header received
0618 22:20:23.620569 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:20:23.620574 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:20:23.620616 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:20:23.621108 (+ 492us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:20:23.621218 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:23.621983 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:23.623028 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:23.623294 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:23.623727 (+ 433us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:23.625421 (+ 1694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:20:23.625424 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:20:23.625427 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:20:23.625449 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:20:23.628171 (+ 2722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:23.628627 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:23.628630 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:23.628632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:23.628710 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:23.628986 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:23.628990 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:23.628991 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:23.629145 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:20:23.629853 (+ 708us) server_negotiation.cc:1035] Waiting for connection context
0618 22:20:23.629973 (+ 120us) server_negotiation.cc:299] Negotiation successful
0618 22:20:23.630086 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":285,"thread_start_us":99,"threads_started":1}
I20260618 22:20:23.630836 13802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:23.618521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:40849 (local address 127.0.0.1:55748)
0618 22:20:23.618690 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:23.618706 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 22:20:23.618780 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:20:23.619205 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:20:23.619208 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:20:23.619379 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:20:23.619468 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:23.619475 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:23.620612 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:23.620616 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:23.621399 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:23.621409 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:23.622024 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:23.622030 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:23.622120 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:23.622811 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:20:23.622826 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:20:23.624262 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:20:23.627789 (+ 3527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:20:23.627792 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:20:23.627794 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:20:23.628089 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:20:23.629281 (+ 1192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:20:23.629284 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:20:23.629285 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:20:23.629378 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:20:23.630370 (+ 992us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:20:23.630373 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:20:23.630467 (+ 94us) client_negotiation.cc:769] Sending connection context
0618 22:20:23.630547 (+ 80us) client_negotiation.cc:240] Negotiation successful
0618 22:20:23.630670 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260618 22:20:23.631884 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:20:23.633293 13950 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5caa1743702b4718b3225cb48dcf31f6: Neither blocks nor log segments found. Creating new log.
I20260618 22:20:23.633548 13874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba7dbd10493740cbaa0190c0f67fb10a: Bootstrap starting.
I20260618 22:20:23.634445 13950 log.cc:824] T 00000000000000000000000000000000 P 5caa1743702b4718b3225cb48dcf31f6: Log is configured to *not* fsync() on all Append() calls
I20260618 22:20:23.635758 13874 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba7dbd10493740cbaa0190c0f67fb10a: Neither blocks nor log segments found. Creating new log.
I20260618 22:20:23.636407 13874 log.cc:824] T 00000000000000000000000000000000 P ba7dbd10493740cbaa0190c0f67fb10a: Log is configured to *not* fsync() on all Append() calls
I20260618 22:20:23.637217 13955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:23.618586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55748 (local address 127.0.98.124:40849)
0618 22:20:23.619007 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0618 22:20:23.619015 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0618 22:20:23.619026 (+ 11us) server_negotiation.cc:407] Connection header received
0618 22:20:23.619056 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:20:23.619058 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:20:23.619102 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:20:23.619189 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:20:23.619599 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:23.620489 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:23.621530 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:23.621882 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:23.622354 (+ 472us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:23.624775 (+ 2421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:20:23.624778 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:20:23.624782 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:20:23.624809 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:20:23.627668 (+ 2859us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:23.628338 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:23.628341 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:23.628342 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:23.628418 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:23.629490 (+ 1072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:23.629493 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:23.629495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:23.630026 (+ 531us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:20:23.636849 (+ 6823us) server_negotiation.cc:1035] Waiting for connection context
0618 22:20:23.636933 (+ 84us) server_negotiation.cc:299] Negotiation successful
0618 22:20:23.637059 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":365,"thread_start_us":112,"threads_started":1}
I20260618 22:20:23.638558 13874 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba7dbd10493740cbaa0190c0f67fb10a: No bootstrap required, opened a new log
I20260618 22:20:23.640147 13799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 518d01444eef4511b90ee405aac52735: Bootstrap starting.
I20260618 22:20:23.641156 13950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5caa1743702b4718b3225cb48dcf31f6: No bootstrap required, opened a new log
I20260618 22:20:23.642630 13803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:23.634093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55460 (local address 127.0.98.126:40419)
0618 22:20:23.634208 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0618 22:20:23.634212 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:20:23.634728 (+ 516us) server_negotiation.cc:407] Connection header received
0618 22:20:23.634836 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:20:23.634839 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:20:23.634876 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:20:23.634965 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:20:23.635595 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:23.636139 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:23.636960 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:23.637268 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:23.637361 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:23.639120 (+ 1759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:20:23.639124 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:20:23.639126 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:20:23.639149 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:20:23.640793 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:23.641283 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:23.641287 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:23.641289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:23.641367 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:23.641678 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:23.641681 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:23.641683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:23.641878 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:20:23.642005 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0618 22:20:23.642357 (+ 352us) server_negotiation.cc:299] Negotiation successful
0618 22:20:23.642483 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260618 22:20:23.642802 13799 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 518d01444eef4511b90ee405aac52735: Neither blocks nor log segments found. Creating new log.
I20260618 22:20:23.643607 13799 log.cc:824] T 00000000000000000000000000000000 P 518d01444eef4511b90ee405aac52735: Log is configured to *not* fsync() on all Append() calls
I20260618 22:20:23.644212 13874 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba7dbd10493740cbaa0190c0f67fb10a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "518d01444eef4511b90ee405aac52735" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40419 } } peers { permanent_uuid: "ba7dbd10493740cbaa0190c0f67fb10a" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 43265 } } peers { permanent_uuid: "5caa1743702b4718b3225cb48dcf31f6" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40849 } }
I20260618 22:20:23.644532 13874 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba7dbd10493740cbaa0190c0f67fb10a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:20:23.644635 13874 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba7dbd10493740cbaa0190c0f67fb10a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba7dbd10493740cbaa0190c0f67fb10a, State: Initialized, Role: FOLLOWER
I20260618 22:20:23.645171 13874 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba7dbd10493740cbaa0190c0f67fb10a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "518d01444eef4511b90ee405aac52735" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40419 } } peers { permanent_uuid: "ba7dbd10493740cbaa0190c0f67fb10a" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 43265 } } peers { permanent_uuid: "5caa1743702b4718b3225cb48dcf31f6" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40849 } }
I20260618 22:20:23.645629 13799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 518d01444eef4511b90ee405aac52735: No bootstrap required, opened a new log
I20260618 22:20:23.646014 13950 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5caa1743702b4718b3225cb48dcf31f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "518d01444eef4511b90ee405aac52735" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40419 } } peers { permanent_uuid: "ba7dbd10493740cbaa0190c0f67fb10a" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 43265 } } peers { permanent_uuid: "5caa1743702b4718b3225cb48dcf31f6" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40849 } }
I20260618 22:20:23.646236 13950 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5caa1743702b4718b3225cb48dcf31f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:20:23.646292 13950 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5caa1743702b4718b3225cb48dcf31f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5caa1743702b4718b3225cb48dcf31f6, State: Initialized, Role: FOLLOWER
I20260618 22:20:23.646773 13874 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba7dbd10493740cbaa0190c0f67fb10a [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:20:23.649432 13967 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba7dbd10493740cbaa0190c0f67fb10a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "518d01444eef4511b90ee405aac52735" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40419 } } peers { permanent_uuid: "ba7dbd10493740cbaa0190c0f67fb10a" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 43265 } } peers { permanent_uuid: "5caa1743702b4718b3225cb48dcf31f6" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40849 } } }
I20260618 22:20:23.649689 13967 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba7dbd10493740cbaa0190c0f67fb10a [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:20:23.650285 13799 raft_consensus.cc:348] T 00000000000000000000000000000000 P 518d01444eef4511b90ee405aac52735 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "518d01444eef4511b90ee405aac52735" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40419 } } peers { permanent_uuid: "ba7dbd10493740cbaa0190c0f67fb10a" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 43265 } } peers { permanent_uuid: "5caa1743702b4718b3225cb48dcf31f6" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40849 } }
I20260618 22:20:23.650606 13950 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5caa1743702b4718b3225cb48dcf31f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "518d01444eef4511b90ee405aac52735" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40419 } } peers { permanent_uuid: "ba7dbd10493740cbaa0190c0f67fb10a" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 43265 } } peers { permanent_uuid: "5caa1743702b4718b3225cb48dcf31f6" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40849 } }
I20260618 22:20:23.652222 13950 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5caa1743702b4718b3225cb48dcf31f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:20:23.652570 13955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:23.638259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55778 (local address 127.0.98.124:40849)
0618 22:20:23.638348 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0618 22:20:23.638352 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:20:23.643371 (+ 5019us) server_negotiation.cc:407] Connection header received
0618 22:20:23.643538 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:20:23.643542 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:20:23.643582 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:20:23.643629 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:20:23.644463 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:23.645246 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:23.646295 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:23.646616 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:23.646691 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:23.648442 (+ 1751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:20:23.648446 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:20:23.648449 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:20:23.648474 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:20:23.650438 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:23.651002 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:23.651006 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:23.651009 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:23.651100 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:23.651432 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:23.651438 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:23.651441 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:23.651723 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:20:23.651857 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0618 22:20:23.652292 (+ 435us) server_negotiation.cc:299] Negotiation successful
0618 22:20:23.652405 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260618 22:20:23.653981 13799 raft_consensus.cc:374] T 00000000000000000000000000000000 P 518d01444eef4511b90ee405aac52735 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:20:23.654039 13880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:23.634142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42764 (local address 127.0.98.125:43265)
0618 22:20:23.634271 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0618 22:20:23.634275 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:20:23.638562 (+ 4287us) server_negotiation.cc:407] Connection header received
0618 22:20:23.638978 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:20:23.638982 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:20:23.639030 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:20:23.639079 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:20:23.646607 (+ 7528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:23.647337 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:23.648480 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:23.648832 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:23.648906 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:23.650576 (+ 1670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:20:23.650580 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:20:23.650583 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:20:23.650608 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:20:23.652320 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:23.652762 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:23.652765 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:23.652767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:23.652850 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:23.653149 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:23.653164 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:23.653165 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:23.653349 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:20:23.653429 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0618 22:20:23.653740 (+ 311us) server_negotiation.cc:299] Negotiation successful
0618 22:20:23.653853 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260618 22:20:23.654297 13799 raft_consensus.cc:729] T 00000000000000000000000000000000 P 518d01444eef4511b90ee405aac52735 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 518d01444eef4511b90ee405aac52735, State: Initialized, Role: FOLLOWER
I20260618 22:20:23.654853 13799 consensus_queue.cc:260] T 00000000000000000000000000000000 P 518d01444eef4511b90ee405aac52735 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "518d01444eef4511b90ee405aac52735" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40419 } } peers { permanent_uuid: "ba7dbd10493740cbaa0190c0f67fb10a" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 43265 } } peers { permanent_uuid: "5caa1743702b4718b3225cb48dcf31f6" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40849 } }
I20260618 22:20:23.655678 13991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 518d01444eef4511b90ee405aac52735 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "518d01444eef4511b90ee405aac52735" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40419 } } peers { permanent_uuid: "ba7dbd10493740cbaa0190c0f67fb10a" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 43265 } } peers { permanent_uuid: "5caa1743702b4718b3225cb48dcf31f6" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40849 } } }
I20260618 22:20:23.655857 13991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 518d01444eef4511b90ee405aac52735 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:20:23.656067 13799 sys_catalog.cc:564] T 00000000000000000000000000000000 P 518d01444eef4511b90ee405aac52735 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:20:23.650694 13806 master_runner.cc:186] Error getting master registration for 127.0.98.126:40419: OK, instance_id { permanent_uuid: "518d01444eef4511b90ee405aac52735" instance_seqno: 1781821223153694 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260618 22:20:23.658118 13806 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260618 22:20:23.658928 13992 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5caa1743702b4718b3225cb48dcf31f6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 22:20:23.658996 13992 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 22:20:23.659152 13979 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5caa1743702b4718b3225cb48dcf31f6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:20:23.659281 13979 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5caa1743702b4718b3225cb48dcf31f6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "518d01444eef4511b90ee405aac52735" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40419 } } peers { permanent_uuid: "ba7dbd10493740cbaa0190c0f67fb10a" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 43265 } } peers { permanent_uuid: "5caa1743702b4718b3225cb48dcf31f6" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40849 } }
I20260618 22:20:23.656417 13881 master_runner.cc:186] Error getting master registration for 127.0.98.126:40419: OK, instance_id { permanent_uuid: "518d01444eef4511b90ee405aac52735" instance_seqno: 1781821223153694 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260618 22:20:23.659540 13881 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260618 22:20:23.660588 13773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5caa1743702b4718b3225cb48dcf31f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "518d01444eef4511b90ee405aac52735" is_pre_election: true
I20260618 22:20:23.660842 13773 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 518d01444eef4511b90ee405aac52735 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5caa1743702b4718b3225cb48dcf31f6 in term 0.
I20260618 22:20:23.661273 13898 leader_election.cc:302] T 00000000000000000000000000000000 P 5caa1743702b4718b3225cb48dcf31f6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 518d01444eef4511b90ee405aac52735, 5caa1743702b4718b3225cb48dcf31f6; no voters:
I20260618 22:20:23.661753 13979 leader_election.cc:288] T 00000000000000000000000000000000 P 5caa1743702b4718b3225cb48dcf31f6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 518d01444eef4511b90ee405aac52735 (127.0.98.126:40419), ba7dbd10493740cbaa0190c0f67fb10a (127.0.98.125:43265)
I20260618 22:20:23.661892 13979 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5caa1743702b4718b3225cb48dcf31f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "518d01444eef4511b90ee405aac52735" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40419 } } peers { permanent_uuid: "ba7dbd10493740cbaa0190c0f67fb10a" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 43265 } } peers { permanent_uuid: "5caa1743702b4718b3225cb48dcf31f6" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40849 } } }
I20260618 22:20:23.662046 13979 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5caa1743702b4718b3225cb48dcf31f6 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:20:23.662153 13979 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5caa1743702b4718b3225cb48dcf31f6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:20:23.662240 13979 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5caa1743702b4718b3225cb48dcf31f6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:20:23.662317 13979 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5caa1743702b4718b3225cb48dcf31f6 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:20:23.662269 13848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5caa1743702b4718b3225cb48dcf31f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba7dbd10493740cbaa0190c0f67fb10a" is_pre_election: true
I20260618 22:20:23.662510 13848 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ba7dbd10493740cbaa0190c0f67fb10a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5caa1743702b4718b3225cb48dcf31f6 in term 0.
I20260618 22:20:23.664023 13979 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5caa1743702b4718b3225cb48dcf31f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "518d01444eef4511b90ee405aac52735" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40419 } } peers { permanent_uuid: "ba7dbd10493740cbaa0190c0f67fb10a" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 43265 } } peers { permanent_uuid: "5caa1743702b4718b3225cb48dcf31f6" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40849 } }
I20260618 22:20:23.664603 13773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5caa1743702b4718b3225cb48dcf31f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "518d01444eef4511b90ee405aac52735"
I20260618 22:20:23.664726 13773 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 518d01444eef4511b90ee405aac52735 [term 0 FOLLOWER]: Advancing to term 1
W20260618 22:20:23.665235 13978 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ba7dbd10493740cbaa0190c0f67fb10a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 22:20:23.665346 13978 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 22:20:23.666028 13731 master_runner.cc:232] Existing masters have differing terms: 0 vs 1
I20260618 22:20:23.666145 13731 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260618 22:20:23.666014 13773 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 518d01444eef4511b90ee405aac52735 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5caa1743702b4718b3225cb48dcf31f6 in term 1.
I20260618 22:20:23.666260 13979 leader_election.cc:288] T 00000000000000000000000000000000 P 5caa1743702b4718b3225cb48dcf31f6 [CANDIDATE]: Term 1 election: Requested vote from peers 518d01444eef4511b90ee405aac52735 (127.0.98.126:40419), ba7dbd10493740cbaa0190c0f67fb10a (127.0.98.125:43265)
I20260618 22:20:23.666498 13898 leader_election.cc:302] T 00000000000000000000000000000000 P 5caa1743702b4718b3225cb48dcf31f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 518d01444eef4511b90ee405aac52735, 5caa1743702b4718b3225cb48dcf31f6; no voters:
I20260618 22:20:23.666627 13848 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5caa1743702b4718b3225cb48dcf31f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba7dbd10493740cbaa0190c0f67fb10a"
I20260618 22:20:23.666759 13848 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba7dbd10493740cbaa0190c0f67fb10a [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:20:23.667958 13848 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ba7dbd10493740cbaa0190c0f67fb10a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5caa1743702b4718b3225cb48dcf31f6 in term 1.
W20260618 22:20:23.669430 14004 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 518d01444eef4511b90ee405aac52735: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 22:20:23.669515 14004 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 22:20:23.669629 13979 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5caa1743702b4718b3225cb48dcf31f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:20:23.669917 13979 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5caa1743702b4718b3225cb48dcf31f6 [term 1 LEADER]: Becoming Leader. State: Replica: 5caa1743702b4718b3225cb48dcf31f6, State: Running, Role: LEADER
I20260618 22:20:23.670281 13979 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5caa1743702b4718b3225cb48dcf31f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "518d01444eef4511b90ee405aac52735" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40419 } } peers { permanent_uuid: "ba7dbd10493740cbaa0190c0f67fb10a" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 43265 } } peers { permanent_uuid: "5caa1743702b4718b3225cb48dcf31f6" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40849 } }
I20260618 22:20:23.672870 13980 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5caa1743702b4718b3225cb48dcf31f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5caa1743702b4718b3225cb48dcf31f6. Latest consensus state: current_term: 1 leader_uuid: "5caa1743702b4718b3225cb48dcf31f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "518d01444eef4511b90ee405aac52735" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40419 } } peers { permanent_uuid: "ba7dbd10493740cbaa0190c0f67fb10a" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 43265 } } peers { permanent_uuid: "5caa1743702b4718b3225cb48dcf31f6" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40849 } } }
I20260618 22:20:23.673085 13980 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5caa1743702b4718b3225cb48dcf31f6 [sys.catalog]: This master's current role is: LEADER
I20260618 22:20:23.678227 14009 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:20:23.679316 14009 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:20:23.690788 13848 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ba7dbd10493740cbaa0190c0f67fb10a [term 1 FOLLOWER]: Refusing update from remote peer 5caa1743702b4718b3225cb48dcf31f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:20:23.690979 13773 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 518d01444eef4511b90ee405aac52735 [term 1 FOLLOWER]: Refusing update from remote peer 5caa1743702b4718b3225cb48dcf31f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:20:23.691532 13980 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5caa1743702b4718b3225cb48dcf31f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba7dbd10493740cbaa0190c0f67fb10a" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 43265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:20:23.692210 13979 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5caa1743702b4718b3225cb48dcf31f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "518d01444eef4511b90ee405aac52735" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:20:23.695729 13991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 518d01444eef4511b90ee405aac52735 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5caa1743702b4718b3225cb48dcf31f6. Latest consensus state: current_term: 1 leader_uuid: "5caa1743702b4718b3225cb48dcf31f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "518d01444eef4511b90ee405aac52735" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40419 } } peers { permanent_uuid: "ba7dbd10493740cbaa0190c0f67fb10a" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 43265 } } peers { permanent_uuid: "5caa1743702b4718b3225cb48dcf31f6" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40849 } } }
I20260618 22:20:23.695833 13991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 518d01444eef4511b90ee405aac52735 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:20:23.700467 13967 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba7dbd10493740cbaa0190c0f67fb10a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5caa1743702b4718b3225cb48dcf31f6. Latest consensus state: current_term: 1 leader_uuid: "5caa1743702b4718b3225cb48dcf31f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "518d01444eef4511b90ee405aac52735" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40419 } } peers { permanent_uuid: "ba7dbd10493740cbaa0190c0f67fb10a" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 43265 } } peers { permanent_uuid: "5caa1743702b4718b3225cb48dcf31f6" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40849 } } }
I20260618 22:20:23.700572 13967 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba7dbd10493740cbaa0190c0f67fb10a [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:20:23.707129 13979 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5caa1743702b4718b3225cb48dcf31f6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5caa1743702b4718b3225cb48dcf31f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "518d01444eef4511b90ee405aac52735" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40419 } } peers { permanent_uuid: "ba7dbd10493740cbaa0190c0f67fb10a" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 43265 } } peers { permanent_uuid: "5caa1743702b4718b3225cb48dcf31f6" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40849 } } }
I20260618 22:20:23.707238 13979 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5caa1743702b4718b3225cb48dcf31f6 [sys.catalog]: This master's current role is: LEADER
I20260618 22:20:23.707626 14010 mvcc.cc:205] Tried to move back new op lower bound from 7298339732230234112 to 7298339732157009920. Current Snapshot: MvccSnapshot[applied={T|T < 7298339732230234112}]
I20260618 22:20:23.710572 13967 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba7dbd10493740cbaa0190c0f67fb10a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5caa1743702b4718b3225cb48dcf31f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "518d01444eef4511b90ee405aac52735" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40419 } } peers { permanent_uuid: "ba7dbd10493740cbaa0190c0f67fb10a" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 43265 } } peers { permanent_uuid: "5caa1743702b4718b3225cb48dcf31f6" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40849 } } }
I20260618 22:20:23.710673 13967 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba7dbd10493740cbaa0190c0f67fb10a [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:20:23.711287 14013 mvcc.cc:205] Tried to move back new op lower bound from 7298339732230234112 to 7298339732157009920. Current Snapshot: MvccSnapshot[applied={T|T < 7298339732230234112}]
I20260618 22:20:23.712191 14006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5caa1743702b4718b3225cb48dcf31f6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5caa1743702b4718b3225cb48dcf31f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "518d01444eef4511b90ee405aac52735" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40419 } } peers { permanent_uuid: "ba7dbd10493740cbaa0190c0f67fb10a" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 43265 } } peers { permanent_uuid: "5caa1743702b4718b3225cb48dcf31f6" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40849 } } }
I20260618 22:20:23.712298 14006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5caa1743702b4718b3225cb48dcf31f6 [sys.catalog]: This master's current role is: LEADER
I20260618 22:20:23.713125 13991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 518d01444eef4511b90ee405aac52735 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5caa1743702b4718b3225cb48dcf31f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "518d01444eef4511b90ee405aac52735" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40419 } } peers { permanent_uuid: "ba7dbd10493740cbaa0190c0f67fb10a" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 43265 } } peers { permanent_uuid: "5caa1743702b4718b3225cb48dcf31f6" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 40849 } } }
I20260618 22:20:23.713227 13991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 518d01444eef4511b90ee405aac52735 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:20:23.714246 14011 mvcc.cc:205] Tried to move back new op lower bound from 7298339732230234112 to 7298339732157009920. Current Snapshot: MvccSnapshot[applied={T|T < 7298339732230234112}]
I20260618 22:20:23.715884 14009 catalog_manager.cc:1347] Generated new cluster ID: 150759b4a1a346468c8e51a662693a54
I20260618 22:20:23.715957 14009 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:20:23.736382 14009 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:20:23.737661 14009 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:20:23.749229 14009 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5caa1743702b4718b3225cb48dcf31f6: Generated new TSK 0
I20260618 22:20:23.751832 14009 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:20:24.666642 13978 catalog_manager.cc:1259] Loaded cluster ID: 150759b4a1a346468c8e51a662693a54
I20260618 22:20:24.666755 13978 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ba7dbd10493740cbaa0190c0f67fb10a: loading cluster ID for follower catalog manager: success
I20260618 22:20:24.668397 13978 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ba7dbd10493740cbaa0190c0f67fb10a: acquiring CA information for follower catalog manager: success
I20260618 22:20:24.669565 13978 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ba7dbd10493740cbaa0190c0f67fb10a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260618 22:20:24.670560 14004 catalog_manager.cc:1259] Loaded cluster ID: 150759b4a1a346468c8e51a662693a54
I20260618 22:20:24.670624 14004 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 518d01444eef4511b90ee405aac52735: loading cluster ID for follower catalog manager: success
I20260618 22:20:24.672075 14004 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 518d01444eef4511b90ee405aac52735: acquiring CA information for follower catalog manager: success
I20260618 22:20:24.673072 14004 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 518d01444eef4511b90ee405aac52735: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260618 22:20:24.823575 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 13731
I20260618 22:20:24.829777 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 13806
I20260618 22:20:24.835243 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 13881
2026-06-18T22:20:24Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3014 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781821065380026-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:20:24 dist-test-slave-g6r4 krb5kdc[14025](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:20:24 dist-test-slave-g6r4 krb5kdc[14025](info): set up 2 sockets
Jun 18 22:20:24 dist-test-slave-g6r4 krb5kdc[14025](info): commencing operation
krb5kdc: starting...
W20260618 22:20:25.861949 393 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781821065380026-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781821065380026-393-0/krb5kdc/test-user.keytab.
Jun 18 22:20:25 dist-test-slave-g6r4 krb5kdc[14025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:20:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:20:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260618 22:20:25.977130 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781821065380026-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781821065380026-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781821065380026-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:36661
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:39825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:36661
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:20:26.083951 14038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:20:26.084178 14038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:20:26.084249 14038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:20:26.088208 14038 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:20:26.088254 14038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:20:26.088274 14038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:20:26.088289 14038 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:20:26.088330 14038 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:20:26.093355 14038 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:39825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781821065380026-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781821065380026-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:36661
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:36661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781821065380026-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:20:26.094259 14038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:20:26.095160 14038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:20:26.103093 14043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:20:26.103092 14046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:20:26.103118 14044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:20:26.103480 14038 server_base.cc:1034] running on GCE node
I20260618 22:20:26.104004 14038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:20:26.104969 14038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:20:26.106153 14038 hybrid_clock.cc:648] HybridClock initialized: now 1781821226106129 us; error 35 us; skew 500 ppm
Jun 18 22:20:26 dist-test-slave-g6r4 krb5kdc[14025](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821226, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:20:26.108677 14038 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260618 22:20:26.109504 14038 webserver.cc:460] Webserver started at http://127.0.98.126:40249/ using document root <none> and password file <none>
I20260618 22:20:26.110083 14038 fs_manager.cc:359] Metadata directory not provided
I20260618 22:20:26.110172 14038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:20:26.110383 14038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:20:26.112272 14038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781821065380026-393-0/minicluster-data/master-0/data/instance:
uuid: "ecd42f4fe8bd47bb9c07b50ea076237a"
format_stamp: "Formatted at 2026-06-18 22:20:26 on dist-test-slave-g6r4"
server_key: "b3d95e9d5d7aad7690ac9986f7f42497"
server_key_iv: "d8409268e0ba713aa80de3901a8c6307"
server_key_version: "encryptionkey@0"
I20260618 22:20:26.112813 14038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781821065380026-393-0/minicluster-data/master-0/wal/instance:
uuid: "ecd42f4fe8bd47bb9c07b50ea076237a"
format_stamp: "Formatted at 2026-06-18 22:20:26 on dist-test-slave-g6r4"
server_key: "b3d95e9d5d7aad7690ac9986f7f42497"
server_key_iv: "d8409268e0ba713aa80de3901a8c6307"
server_key_version: "encryptionkey@0"
I20260618 22:20:26.116202 14038 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260618 22:20:26.118491 14053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:20:26.119623 14038 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:20:26.119761 14038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781821065380026-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781821065380026-393-0/minicluster-data/master-0/wal
uuid: "ecd42f4fe8bd47bb9c07b50ea076237a"
format_stamp: "Formatted at 2026-06-18 22:20:26 on dist-test-slave-g6r4"
server_key: "b3d95e9d5d7aad7690ac9986f7f42497"
server_key_iv: "d8409268e0ba713aa80de3901a8c6307"
server_key_version: "encryptionkey@0"
I20260618 22:20:26.119864 14038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781821065380026-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781821065380026-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781821065380026-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:20:26.136009 14038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:20:26.138605 14038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:20:26.138794 14038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:20:26.145417 14038 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:36661
I20260618 22:20:26.145424 14105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:36661 every 8 connection(s)
I20260618 22:20:26.146330 14038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
I20260618 22:20:26.148778 14106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:20:26.152688 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 14038
I20260618 22:20:26.152809 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781821065380026-393-0/minicluster-data/master-0/wal/instance
I20260618 22:20:26.153116 393 external_mini_cluster.cc:1428] Setting key 99f374b77750875cba86b3acddde0ebd
I20260618 22:20:26.154311 14106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecd42f4fe8bd47bb9c07b50ea076237a: Bootstrap starting.
I20260618 22:20:26.156522 14106 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ecd42f4fe8bd47bb9c07b50ea076237a: Neither blocks nor log segments found. Creating new log.
I20260618 22:20:26.157294 14106 log.cc:824] T 00000000000000000000000000000000 P ecd42f4fe8bd47bb9c07b50ea076237a: Log is configured to *not* fsync() on all Append() calls
I20260618 22:20:26.159230 14106 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ecd42f4fe8bd47bb9c07b50ea076237a: No bootstrap required, opened a new log
Jun 18 22:20:26 dist-test-slave-g6r4 krb5kdc[14025](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:20:26.163774 14106 raft_consensus.cc:348] T 00000000000000000000000000000000 P ecd42f4fe8bd47bb9c07b50ea076237a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecd42f4fe8bd47bb9c07b50ea076237a" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36661 } }
I20260618 22:20:26.164016 14106 raft_consensus.cc:374] T 00000000000000000000000000000000 P ecd42f4fe8bd47bb9c07b50ea076237a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:20:26.164084 14106 raft_consensus.cc:729] T 00000000000000000000000000000000 P ecd42f4fe8bd47bb9c07b50ea076237a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecd42f4fe8bd47bb9c07b50ea076237a, State: Initialized, Role: FOLLOWER
I20260618 22:20:26.164515 14106 consensus_queue.cc:260] T 00000000000000000000000000000000 P ecd42f4fe8bd47bb9c07b50ea076237a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecd42f4fe8bd47bb9c07b50ea076237a" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36661 } }
I20260618 22:20:26.164669 14106 raft_consensus.cc:388] T 00000000000000000000000000000000 P ecd42f4fe8bd47bb9c07b50ea076237a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:20:26.164739 14106 raft_consensus.cc:482] T 00000000000000000000000000000000 P ecd42f4fe8bd47bb9c07b50ea076237a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:20:26.164848 14106 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ecd42f4fe8bd47bb9c07b50ea076237a [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:20:26.165743 14106 raft_consensus.cc:504] T 00000000000000000000000000000000 P ecd42f4fe8bd47bb9c07b50ea076237a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecd42f4fe8bd47bb9c07b50ea076237a" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36661 } }
I20260618 22:20:26.166126 14106 leader_election.cc:302] T 00000000000000000000000000000000 P ecd42f4fe8bd47bb9c07b50ea076237a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ecd42f4fe8bd47bb9c07b50ea076237a; no voters:
I20260618 22:20:26.166448 14106 leader_election.cc:288] T 00000000000000000000000000000000 P ecd42f4fe8bd47bb9c07b50ea076237a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:20:26.166548 14111 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ecd42f4fe8bd47bb9c07b50ea076237a [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:20:26.166765 14111 raft_consensus.cc:686] T 00000000000000000000000000000000 P ecd42f4fe8bd47bb9c07b50ea076237a [term 1 LEADER]: Becoming Leader. State: Replica: ecd42f4fe8bd47bb9c07b50ea076237a, State: Running, Role: LEADER
I20260618 22:20:26.167100 14111 consensus_queue.cc:237] T 00000000000000000000000000000000 P ecd42f4fe8bd47bb9c07b50ea076237a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecd42f4fe8bd47bb9c07b50ea076237a" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36661 } }
I20260618 22:20:26.167537 14109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:26.154334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46504 (local address 127.0.98.126:36661)
0618 22:20:26.154759 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0618 22:20:26.154776 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0618 22:20:26.154808 (+ 32us) server_negotiation.cc:407] Connection header received
0618 22:20:26.155424 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:20:26.155440 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:20:26.155749 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:20:26.156011 (+ 262us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:20:26.156960 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:26.157999 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:26.159096 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:26.159598 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:26.159716 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:26.162354 (+ 2638us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:20:26.162363 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:20:26.162382 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:20:26.162412 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:20:26.164260 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:26.164708 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:26.164714 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:26.164719 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:26.164906 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:26.165227 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:26.165231 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:26.165233 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:26.165598 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:20:26.165744 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0618 22:20:26.166147 (+ 403us) server_negotiation.cc:299] Negotiation successful
0618 22:20:26.166627 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":130,"threads_started":1}
I20260618 22:20:26.167538 14106 sys_catalog.cc:564] T 00000000000000000000000000000000 P ecd42f4fe8bd47bb9c07b50ea076237a [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:20:26.169160 14112 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecd42f4fe8bd47bb9c07b50ea076237a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ecd42f4fe8bd47bb9c07b50ea076237a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecd42f4fe8bd47bb9c07b50ea076237a" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36661 } } }
I20260618 22:20:26.169459 14113 sys_catalog.cc:455] T 00000000000000000000000000000000 P ecd42f4fe8bd47bb9c07b50ea076237a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ecd42f4fe8bd47bb9c07b50ea076237a. Latest consensus state: current_term: 1 leader_uuid: "ecd42f4fe8bd47bb9c07b50ea076237a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ecd42f4fe8bd47bb9c07b50ea076237a" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36661 } } }
I20260618 22:20:26.169613 14113 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecd42f4fe8bd47bb9c07b50ea076237a [sys.catalog]: This master's current role is: LEADER
I20260618 22:20:26.169649 14112 sys_catalog.cc:458] T 00000000000000000000000000000000 P ecd42f4fe8bd47bb9c07b50ea076237a [sys.catalog]: This master's current role is: LEADER
I20260618 22:20:26.170907 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260618 22:20:26.172746 14126 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ecd42f4fe8bd47bb9c07b50ea076237a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 22:20:26.172824 14126 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 22:20:26.172902 14125 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:20:26.173580 14125 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:20:26.178264 14125 catalog_manager.cc:1347] Generated new cluster ID: 56d4ee0e4394468ebfb94aa0a1f52d09
I20260618 22:20:26.178329 14125 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:20:26.192337 14125 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:20:26.193125 14125 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:20:26.198149 14125 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ecd42f4fe8bd47bb9c07b50ea076237a: Generated new TSK 0
I20260618 22:20:26.198710 14125 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:20:26.226367 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 14038
2026-06-18T22:20:26Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1389 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:20:26 dist-test-slave-g6r4 krb5kdc[14133](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:20:26 dist-test-slave-g6r4 krb5kdc[14133](info): set up 2 sockets
Jun 18 22:20:26 dist-test-slave-g6r4 krb5kdc[14133](info): commencing operation
krb5kdc: starting...
W20260618 22:20:27.298030 393 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/krb5kdc/test-user.keytab.
Jun 18 22:20:27 dist-test-slave-g6r4 krb5kdc[14133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:20:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:20:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260618 22:20:27.412919 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:38557
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:43207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:38557
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:20:27.518505 14146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:20:27.518765 14146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:20:27.518836 14146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:20:27.522737 14146 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:20:27.522815 14146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:20:27.522855 14146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:20:27.522914 14146 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:20:27.522953 14146 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:20:27.527900 14146 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:43207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:38557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:38557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:20:27.528950 14146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:20:27.530203 14146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:20:27.538141 14151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:20:27.538174 14146 server_base.cc:1034] running on GCE node
W20260618 22:20:27.538128 14154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:20:27.538146 14152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:20:27.538856 14146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:20:27.539844 14146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:20:27.540995 14146 hybrid_clock.cc:648] HybridClock initialized: now 1781821227540971 us; error 38 us; skew 500 ppm
Jun 18 22:20:27 dist-test-slave-g6r4 krb5kdc[14133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821227, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:20:27.547580 14146 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260618 22:20:27.548381 14146 webserver.cc:460] Webserver started at http://127.0.98.126:44585/ using document root <none> and password file <none>
I20260618 22:20:27.548911 14146 fs_manager.cc:359] Metadata directory not provided
I20260618 22:20:27.549001 14146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:20:27.549229 14146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:20:27.551193 14146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/master-0/data/instance:
uuid: "dbe151a423ee4337936f2226d65741da"
format_stamp: "Formatted at 2026-06-18 22:20:27 on dist-test-slave-g6r4"
server_key: "31a3e970244938294a9a6f0a346c5376"
server_key_iv: "ae14f4c650bcd0940dbaee4523a3337b"
server_key_version: "encryptionkey@0"
I20260618 22:20:27.551752 14146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/master-0/wal/instance:
uuid: "dbe151a423ee4337936f2226d65741da"
format_stamp: "Formatted at 2026-06-18 22:20:27 on dist-test-slave-g6r4"
server_key: "31a3e970244938294a9a6f0a346c5376"
server_key_iv: "ae14f4c650bcd0940dbaee4523a3337b"
server_key_version: "encryptionkey@0"
I20260618 22:20:27.555154 14146 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:20:27.557377 14161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:20:27.558594 14146 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260618 22:20:27.558729 14146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/master-0/wal
uuid: "dbe151a423ee4337936f2226d65741da"
format_stamp: "Formatted at 2026-06-18 22:20:27 on dist-test-slave-g6r4"
server_key: "31a3e970244938294a9a6f0a346c5376"
server_key_iv: "ae14f4c650bcd0940dbaee4523a3337b"
server_key_version: "encryptionkey@0"
I20260618 22:20:27.558833 14146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:20:27.569741 14146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:20:27.572525 14146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:20:27.572727 14146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:20:27.579592 14146 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:38557
I20260618 22:20:27.579923 14213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:38557 every 8 connection(s)
I20260618 22:20:27.580474 14146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
I20260618 22:20:27.583186 14214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:20:27.588852 14214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbe151a423ee4337936f2226d65741da: Bootstrap starting.
I20260618 22:20:27.589692 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 14146
I20260618 22:20:27.589857 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/master-0/wal/instance
I20260618 22:20:27.590243 393 external_mini_cluster.cc:1428] Setting key 1b89c35a0e63120360b045201e46795c
I20260618 22:20:27.591852 14214 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dbe151a423ee4337936f2226d65741da: Neither blocks nor log segments found. Creating new log.
I20260618 22:20:27.592581 14214 log.cc:824] T 00000000000000000000000000000000 P dbe151a423ee4337936f2226d65741da: Log is configured to *not* fsync() on all Append() calls
I20260618 22:20:27.594692 14214 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbe151a423ee4337936f2226d65741da: No bootstrap required, opened a new log
Jun 18 22:20:27 dist-test-slave-g6r4 krb5kdc[14133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:20:27.599454 14214 raft_consensus.cc:348] T 00000000000000000000000000000000 P dbe151a423ee4337936f2226d65741da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbe151a423ee4337936f2226d65741da" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38557 } }
I20260618 22:20:27.599740 14214 raft_consensus.cc:374] T 00000000000000000000000000000000 P dbe151a423ee4337936f2226d65741da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:20:27.599817 14214 raft_consensus.cc:729] T 00000000000000000000000000000000 P dbe151a423ee4337936f2226d65741da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbe151a423ee4337936f2226d65741da, State: Initialized, Role: FOLLOWER
I20260618 22:20:27.600358 14214 consensus_queue.cc:260] T 00000000000000000000000000000000 P dbe151a423ee4337936f2226d65741da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbe151a423ee4337936f2226d65741da" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38557 } }
I20260618 22:20:27.600545 14214 raft_consensus.cc:388] T 00000000000000000000000000000000 P dbe151a423ee4337936f2226d65741da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:20:27.600630 14214 raft_consensus.cc:482] T 00000000000000000000000000000000 P dbe151a423ee4337936f2226d65741da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:20:27.600759 14214 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dbe151a423ee4337936f2226d65741da [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:20:27.601720 14214 raft_consensus.cc:504] T 00000000000000000000000000000000 P dbe151a423ee4337936f2226d65741da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbe151a423ee4337936f2226d65741da" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38557 } }
I20260618 22:20:27.602140 14214 leader_election.cc:302] T 00000000000000000000000000000000 P dbe151a423ee4337936f2226d65741da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dbe151a423ee4337936f2226d65741da; no voters:
I20260618 22:20:27.602516 14214 leader_election.cc:288] T 00000000000000000000000000000000 P dbe151a423ee4337936f2226d65741da [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:20:27.602587 14219 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dbe151a423ee4337936f2226d65741da [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:20:27.602807 14219 raft_consensus.cc:686] T 00000000000000000000000000000000 P dbe151a423ee4337936f2226d65741da [term 1 LEADER]: Becoming Leader. State: Replica: dbe151a423ee4337936f2226d65741da, State: Running, Role: LEADER
I20260618 22:20:27.603178 14219 consensus_queue.cc:237] T 00000000000000000000000000000000 P dbe151a423ee4337936f2226d65741da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbe151a423ee4337936f2226d65741da" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38557 } }
I20260618 22:20:27.603883 14214 sys_catalog.cc:564] T 00000000000000000000000000000000 P dbe151a423ee4337936f2226d65741da [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:20:27.605001 14221 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbe151a423ee4337936f2226d65741da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dbe151a423ee4337936f2226d65741da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbe151a423ee4337936f2226d65741da" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38557 } } }
I20260618 22:20:27.605098 14217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:27.591330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45126 (local address 127.0.98.126:38557)
0618 22:20:27.591708 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0618 22:20:27.591729 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0618 22:20:27.591762 (+ 33us) server_negotiation.cc:407] Connection header received
0618 22:20:27.592391 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:20:27.592408 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:20:27.592730 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:20:27.593063 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:20:27.593875 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:27.594911 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:27.596174 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:27.596655 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:27.596770 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:27.599665 (+ 2895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:20:27.599672 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:20:27.599688 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:20:27.599724 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:20:27.601551 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:27.602102 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:27.602106 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:27.602110 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:27.602220 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:27.602556 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:27.602560 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:27.602564 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:27.603111 (+ 547us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:20:27.603307 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0618 22:20:27.603876 (+ 569us) server_negotiation.cc:299] Negotiation successful
0618 22:20:27.604174 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":123,"threads_started":1}
I20260618 22:20:27.605124 14221 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbe151a423ee4337936f2226d65741da [sys.catalog]: This master's current role is: LEADER
I20260618 22:20:27.606171 14228 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:20:27.606408 14220 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbe151a423ee4337936f2226d65741da [sys.catalog]: SysCatalogTable state changed. Reason: New leader dbe151a423ee4337936f2226d65741da. Latest consensus state: current_term: 1 leader_uuid: "dbe151a423ee4337936f2226d65741da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbe151a423ee4337936f2226d65741da" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38557 } } }
I20260618 22:20:27.606734 14220 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbe151a423ee4337936f2226d65741da [sys.catalog]: This master's current role is: LEADER
I20260618 22:20:27.609400 14228 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:20:27.614437 14228 catalog_manager.cc:1347] Generated new cluster ID: 42893856c82e4dcbb1a581319025cfb1
I20260618 22:20:27.614498 14228 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:20:27.632536 14228 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:20:27.633875 14228 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:20:27.643821 14228 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dbe151a423ee4337936f2226d65741da: Generated new TSK 0
I20260618 22:20:27.644665 14228 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab.
I20260618 22:20:27.687014 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:38557
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:43207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:20:27.801385 14240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:20:27.801613 14240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:20:27.801653 14240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:20:27.805721 14240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:20:27.805773 14240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:20:27.805881 14240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260618 22:20:27.810848 14240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:43207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:38557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:20:27.811721 14240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:20:27.812650 14240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:20:27.821616 14245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:20:27.821626 14248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:20:27.821722 14246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:20:27.821770 14240 server_base.cc:1034] running on GCE node
I20260618 22:20:27.822315 14240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:20:27.822913 14240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:20:27.824079 14240 hybrid_clock.cc:648] HybridClock initialized: now 1781821227824064 us; error 24 us; skew 500 ppm
Jun 18 22:20:27 dist-test-slave-g6r4 krb5kdc[14133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821227, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:20:27.826890 14240 init.cc:377] Logged in from keytab as kudu/127.0.98.65@KRBTEST.COM (short username kudu)
I20260618 22:20:27.827642 14240 webserver.cc:460] Webserver started at http://127.0.98.65:45317/ using document root <none> and password file <none>
I20260618 22:20:27.828122 14240 fs_manager.cc:359] Metadata directory not provided
I20260618 22:20:27.828179 14240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:20:27.828337 14240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:20:27.830247 14240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-0/data/instance:
uuid: "a2c7bdc59554485ba027d28fee230475"
format_stamp: "Formatted at 2026-06-18 22:20:27 on dist-test-slave-g6r4"
server_key: "9a934605ba3da4a73d2275f4cb40e86e"
server_key_iv: "3d1359fb93e157e37f591124f86586f5"
server_key_version: "encryptionkey@0"
I20260618 22:20:27.830732 14240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-0/wal/instance:
uuid: "a2c7bdc59554485ba027d28fee230475"
format_stamp: "Formatted at 2026-06-18 22:20:27 on dist-test-slave-g6r4"
server_key: "9a934605ba3da4a73d2275f4cb40e86e"
server_key_iv: "3d1359fb93e157e37f591124f86586f5"
server_key_version: "encryptionkey@0"
I20260618 22:20:27.834363 14240 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:20:27.836661 14255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:20:27.838150 14240 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:20:27.838263 14240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-0/wal
uuid: "a2c7bdc59554485ba027d28fee230475"
format_stamp: "Formatted at 2026-06-18 22:20:27 on dist-test-slave-g6r4"
server_key: "9a934605ba3da4a73d2275f4cb40e86e"
server_key_iv: "3d1359fb93e157e37f591124f86586f5"
server_key_version: "encryptionkey@0"
I20260618 22:20:27.838384 14240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:20:27.848376 14240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:20:27.851373 14240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:20:27.851567 14240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:20:27.852087 14240 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:20:27.852816 14240 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:20:27.852898 14240 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:20:27.852963 14240 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:20:27.852994 14240 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:20:27.861541 14240 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:34547
I20260618 22:20:27.861588 14368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:34547 every 8 connection(s)
I20260618 22:20:27.862545 14240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
I20260618 22:20:27.863189 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 14240
I20260618 22:20:27.863343 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-0/wal/instance
I20260618 22:20:27.863636 393 external_mini_cluster.cc:1428] Setting key b0b96c2f90178e8d17085fdee16ac244
Jun 18 22:20:27 dist-test-slave-g6r4 krb5kdc[14133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821227, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.66@KRBTEST.COM" created.
I20260618 22:20:27.876992 14217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:27.864398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:57369 (local address 127.0.98.126:38557)
0618 22:20:27.864532 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0618 22:20:27.864536 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:20:27.865464 (+ 928us) server_negotiation.cc:407] Connection header received
0618 22:20:27.867267 (+ 1803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:20:27.867273 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:20:27.867332 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:20:27.867428 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:20:27.868456 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:27.869381 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:27.870566 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:27.871009 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:27.871116 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:27.873689 (+ 2573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:20:27.873692 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:20:27.873694 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:20:27.873716 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:20:27.875023 (+ 1307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:27.875497 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:27.875503 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:27.875507 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:27.875586 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:27.875918 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:27.875921 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:27.875923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:27.876100 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:20:27.876234 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0618 22:20:27.876694 (+ 460us) server_negotiation.cc:299] Negotiation successful
0618 22:20:27.876808 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260618 22:20:27.877713 14371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:27.864770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38557 (local address 127.0.98.65:57369)
0618 22:20:27.865290 (+ 520us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:27.865344 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0618 22:20:27.867066 (+ 1722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:20:27.867539 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:20:27.867545 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:20:27.867877 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:20:27.868312 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:27.868325 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:27.869525 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:27.869528 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:27.870444 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:27.870453 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:27.871152 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:27.871155 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:27.871281 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:27.871870 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:20:27.871885 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:20:27.873526 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:20:27.875137 (+ 1611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:20:27.875143 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:20:27.875156 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:20:27.875396 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:20:27.875680 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:20:27.875684 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:20:27.875686 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:20:27.875818 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:20:27.876232 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:20:27.876239 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:20:27.876467 (+ 228us) client_negotiation.cc:769] Sending connection context
0618 22:20:27.876652 (+ 185us) client_negotiation.cc:240] Negotiation successful
0618 22:20:27.876968 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":141,"threads_started":1}
I20260618 22:20:27.878947 14369 heartbeater.cc:347] Connected to a master server at 127.0.98.126:38557
I20260618 22:20:27.879189 14369 heartbeater.cc:464] Registering TS with master...
I20260618 22:20:27.879721 14369 heartbeater.cc:511] Master 127.0.98.126:38557 requested a full tablet report, sending...
I20260618 22:20:27.881227 14178 ts_manager.cc:194] Registered new tserver with Master: a2c7bdc59554485ba027d28fee230475 (127.0.98.65:34547)
I20260618 22:20:27.882336 14178 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.65@KRBTEST.COM'} at 127.0.98.65:57369
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab.
I20260618 22:20:27.890842 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:38557
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:43207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:20:28.001389 14374 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:20:28.001674 14374 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:20:28.001750 14374 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:20:28.006007 14374 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:20:28.006084 14374 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:20:28.006168 14374 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260618 22:20:28.011227 14374 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:43207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:38557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14374
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:20:28.012212 14374 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:20:28.013185 14374 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:20:28.022346 14382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:20:28.022466 14374 server_base.cc:1034] running on GCE node
W20260618 22:20:28.022373 14379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:20:28.022401 14380 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:20:28.022980 14374 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:20:28.023572 14374 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:20:28.024727 14374 hybrid_clock.cc:648] HybridClock initialized: now 1781821228024712 us; error 76 us; skew 500 ppm
Jun 18 22:20:28 dist-test-slave-g6r4 krb5kdc[14133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821228, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:20:28.027374 14374 init.cc:377] Logged in from keytab as kudu/127.0.98.66@KRBTEST.COM (short username kudu)
I20260618 22:20:28.028201 14374 webserver.cc:460] Webserver started at http://127.0.98.66:33247/ using document root <none> and password file <none>
I20260618 22:20:28.028733 14374 fs_manager.cc:359] Metadata directory not provided
I20260618 22:20:28.028826 14374 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:20:28.029055 14374 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:20:28.031001 14374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-1/data/instance:
uuid: "59d6036ca6d44f6fb120d8c00a3c3665"
format_stamp: "Formatted at 2026-06-18 22:20:28 on dist-test-slave-g6r4"
server_key: "7cf93c9fd250cd36e81ba505b89d34a6"
server_key_iv: "7564773e47f86984d2b4d5a87d2729ed"
server_key_version: "encryptionkey@0"
I20260618 22:20:28.031563 14374 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-1/wal/instance:
uuid: "59d6036ca6d44f6fb120d8c00a3c3665"
format_stamp: "Formatted at 2026-06-18 22:20:28 on dist-test-slave-g6r4"
server_key: "7cf93c9fd250cd36e81ba505b89d34a6"
server_key_iv: "7564773e47f86984d2b4d5a87d2729ed"
server_key_version: "encryptionkey@0"
I20260618 22:20:28.034986 14374 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:20:28.037128 14389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:20:28.038269 14374 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:20:28.038395 14374 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-1/wal
uuid: "59d6036ca6d44f6fb120d8c00a3c3665"
format_stamp: "Formatted at 2026-06-18 22:20:28 on dist-test-slave-g6r4"
server_key: "7cf93c9fd250cd36e81ba505b89d34a6"
server_key_iv: "7564773e47f86984d2b4d5a87d2729ed"
server_key_version: "encryptionkey@0"
I20260618 22:20:28.038497 14374 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:20:28.060328 14374 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:20:28.063616 14374 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:20:28.063799 14374 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:20:28.064374 14374 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:20:28.065128 14374 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:20:28.065172 14374 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:20:28.065236 14374 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:20:28.065271 14374 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:20:28.073489 14374 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:46349
I20260618 22:20:28.073557 14502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:46349 every 8 connection(s)
I20260618 22:20:28.074371 14374 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
I20260618 22:20:28.076422 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 14374
I20260618 22:20:28.076552 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-1/wal/instance
I20260618 22:20:28.076862 393 external_mini_cluster.cc:1428] Setting key 56d316b5f87ae71cc2318f2f92b71e8c
Jun 18 22:20:28 dist-test-slave-g6r4 krb5kdc[14133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821228, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.67@KRBTEST.COM" created.
I20260618 22:20:28.088351 14217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:28.075989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:36713 (local address 127.0.98.126:38557)
0618 22:20:28.076120 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0618 22:20:28.076123 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:20:28.077045 (+ 922us) server_negotiation.cc:407] Connection header received
0618 22:20:28.078536 (+ 1491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:20:28.078539 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:20:28.078590 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:20:28.078672 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:20:28.079944 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:28.080455 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:28.081633 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:28.081857 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:28.082044 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:28.084526 (+ 2482us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:20:28.084529 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:20:28.084532 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:20:28.084564 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:20:28.086069 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:28.086687 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:28.086693 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:28.086694 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:28.086777 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:28.087111 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:28.087117 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:28.087121 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:28.087349 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:20:28.087470 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0618 22:20:28.088060 (+ 590us) server_negotiation.cc:299] Negotiation successful
0618 22:20:28.088185 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260618 22:20:28.089094 14505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:28.076397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38557 (local address 127.0.98.66:36713)
0618 22:20:28.076861 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:28.076902 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0618 22:20:28.078339 (+ 1437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:20:28.078852 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:20:28.078858 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:20:28.079193 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:20:28.079782 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:28.079803 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:28.080583 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:28.080589 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:28.081496 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:28.081505 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:28.082073 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:28.082077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:28.082250 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:28.082806 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:20:28.082822 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:20:28.084369 (+ 1547us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:20:28.086224 (+ 1855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:20:28.086233 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:20:28.086248 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:20:28.086566 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:20:28.086879 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:20:28.086882 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:20:28.086884 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:20:28.087016 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:20:28.087478 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:20:28.087486 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:20:28.087807 (+ 321us) client_negotiation.cc:769] Sending connection context
0618 22:20:28.088010 (+ 203us) client_negotiation.cc:240] Negotiation successful
0618 22:20:28.088302 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"thread_start_us":142,"threads_started":1}
I20260618 22:20:28.090255 14503 heartbeater.cc:347] Connected to a master server at 127.0.98.126:38557
I20260618 22:20:28.090510 14503 heartbeater.cc:464] Registering TS with master...
I20260618 22:20:28.090973 14503 heartbeater.cc:511] Master 127.0.98.126:38557 requested a full tablet report, sending...
I20260618 22:20:28.092094 14178 ts_manager.cc:194] Registered new tserver with Master: 59d6036ca6d44f6fb120d8c00a3c3665 (127.0.98.66:46349)
I20260618 22:20:28.092687 14178 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.66@KRBTEST.COM'} at 127.0.98.66:36713
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab.
I20260618 22:20:28.104382 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:38557
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:43207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:20:28.212646 14508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:20:28.212893 14508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:20:28.212936 14508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:20:28.216769 14508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:20:28.216823 14508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:20:28.216881 14508 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260618 22:20:28.222030 14508 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:43207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:38557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14508
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:20:28.222930 14508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:20:28.223838 14508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:20:28.233306 14513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:20:28.233477 14514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:20:28.233628 14508 server_base.cc:1034] running on GCE node
W20260618 22:20:28.233493 14516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:20:28.234105 14508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:20:28.234736 14508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:20:28.235916 14508 hybrid_clock.cc:648] HybridClock initialized: now 1781821228235901 us; error 27 us; skew 500 ppm
Jun 18 22:20:28 dist-test-slave-g6r4 krb5kdc[14133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821228, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:20:28.238500 14508 init.cc:377] Logged in from keytab as kudu/127.0.98.67@KRBTEST.COM (short username kudu)
I20260618 22:20:28.239295 14508 webserver.cc:460] Webserver started at http://127.0.98.67:37883/ using document root <none> and password file <none>
I20260618 22:20:28.239821 14508 fs_manager.cc:359] Metadata directory not provided
I20260618 22:20:28.239878 14508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:20:28.240036 14508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:20:28.242024 14508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-2/data/instance:
uuid: "222f7b84f00d4e38afcb56c5cc3a1455"
format_stamp: "Formatted at 2026-06-18 22:20:28 on dist-test-slave-g6r4"
server_key: "8b92e37e1fdb89f9145107e6724a1e83"
server_key_iv: "d1ba7ee082abf672580920f1705806b0"
server_key_version: "encryptionkey@0"
I20260618 22:20:28.242573 14508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-2/wal/instance:
uuid: "222f7b84f00d4e38afcb56c5cc3a1455"
format_stamp: "Formatted at 2026-06-18 22:20:28 on dist-test-slave-g6r4"
server_key: "8b92e37e1fdb89f9145107e6724a1e83"
server_key_iv: "d1ba7ee082abf672580920f1705806b0"
server_key_version: "encryptionkey@0"
I20260618 22:20:28.245991 14508 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:20:28.248206 14523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:20:28.249326 14508 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:20:28.249452 14508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-2/wal
uuid: "222f7b84f00d4e38afcb56c5cc3a1455"
format_stamp: "Formatted at 2026-06-18 22:20:28 on dist-test-slave-g6r4"
server_key: "8b92e37e1fdb89f9145107e6724a1e83"
server_key_iv: "d1ba7ee082abf672580920f1705806b0"
server_key_version: "encryptionkey@0"
I20260618 22:20:28.249557 14508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:20:28.260399 14508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:20:28.263152 14508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:20:28.263300 14508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:20:28.263760 14508 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:20:28.264493 14508 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:20:28.264542 14508 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:20:28.264581 14508 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:20:28.264606 14508 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:20:28.273689 14508 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:43361
I20260618 22:20:28.273710 14636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:43361 every 8 connection(s)
I20260618 22:20:28.274582 14508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
I20260618 22:20:28.280381 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 14508
I20260618 22:20:28.280640 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-2/wal/instance
I20260618 22:20:28.280905 393 external_mini_cluster.cc:1428] Setting key a1b8c95435f1a3d33e7b2dcc586034a9
Jun 18 22:20:28 dist-test-slave-g6r4 krb5kdc[14133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821228, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:20:28.287727 14217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:28.276320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:48473 (local address 127.0.98.126:38557)
0618 22:20:28.276471 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0618 22:20:28.276476 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:20:28.277314 (+ 838us) server_negotiation.cc:407] Connection header received
0618 22:20:28.278581 (+ 1267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:20:28.278584 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:20:28.278629 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:20:28.278702 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:20:28.279777 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:28.280310 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:28.281622 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:28.281841 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:28.281970 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:28.284267 (+ 2297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:20:28.284272 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:20:28.284279 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:20:28.284305 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:20:28.285692 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:28.286255 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:28.286260 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:28.286262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:28.286341 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:28.286682 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:28.286687 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:28.286689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:28.286912 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:20:28.287011 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0618 22:20:28.287457 (+ 446us) server_negotiation.cc:299] Negotiation successful
0618 22:20:28.287567 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260618 22:20:28.288452 14639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:28.276712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38557 (local address 127.0.98.67:48473)
0618 22:20:28.277153 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:28.277196 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0618 22:20:28.278393 (+ 1197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:20:28.278834 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:20:28.278841 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:20:28.279179 (+ 338us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:20:28.279649 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:28.279662 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:28.280450 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:28.280458 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:28.281481 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:28.281491 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:28.282013 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:28.282017 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:28.282184 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:28.282755 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:20:28.282770 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:20:28.284108 (+ 1338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:20:28.285826 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:20:28.285832 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:20:28.285845 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:20:28.286152 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:20:28.286439 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:20:28.286442 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:20:28.286444 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:20:28.286586 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:20:28.287007 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:20:28.287013 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:20:28.287237 (+ 224us) client_negotiation.cc:769] Sending connection context
0618 22:20:28.287417 (+ 180us) client_negotiation.cc:240] Negotiation successful
0618 22:20:28.287710 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":114,"threads_started":1}
I20260618 22:20:28.289603 14637 heartbeater.cc:347] Connected to a master server at 127.0.98.126:38557
I20260618 22:20:28.289842 14637 heartbeater.cc:464] Registering TS with master...
I20260618 22:20:28.290313 14637 heartbeater.cc:511] Master 127.0.98.126:38557 requested a full tablet report, sending...
I20260618 22:20:28.291239 14178 ts_manager.cc:194] Registered new tserver with Master: 222f7b84f00d4e38afcb56c5cc3a1455 (127.0.98.67:43361)
I20260618 22:20:28.291791 14178 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.67@KRBTEST.COM'} at 127.0.98.67:48473
I20260618 22:20:28.294580 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 22:20:28.303886 14217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:28.296419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45140 (local address 127.0.98.126:38557)
0618 22:20:28.296522 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0618 22:20:28.296525 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:20:28.296767 (+ 242us) server_negotiation.cc:407] Connection header received
0618 22:20:28.296911 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:20:28.296914 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:20:28.296953 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:20:28.297052 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:20:28.297740 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:28.298296 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:28.299124 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:28.299352 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:28.299458 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:28.300765 (+ 1307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:20:28.300769 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:20:28.300772 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:20:28.300795 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:20:28.302244 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:28.302657 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:28.302662 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:28.302664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:28.302740 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:28.303015 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:28.303018 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:28.303020 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:28.303216 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:20:28.303355 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0618 22:20:28.303586 (+ 231us) server_negotiation.cc:299] Negotiation successful
0618 22:20:28.303688 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260618 22:20:28.306893 14178 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45140:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 22:20:28.310057 14178 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:20:28.324234 14650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:28.317992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55722 (local address 127.0.98.65:34547)
0618 22:20:28.318400 (+ 408us) server_negotiation.cc:206] Beginning negotiation
0618 22:20:28.318412 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0618 22:20:28.318968 (+ 556us) server_negotiation.cc:407] Connection header received
0618 22:20:28.319154 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:20:28.319160 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:20:28.319418 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:20:28.319586 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:20:28.320038 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:28.321220 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:28.322865 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:28.323425 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:28.323488 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:28.323629 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0618 22:20:28.323969 (+ 340us) server_negotiation.cc:299] Negotiation successful
0618 22:20:28.324069 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":98,"threads_started":1}
I20260618 22:20:28.324224 14651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:28.317861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:34547 (local address 127.0.0.1:55722)
0618 22:20:28.318837 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:28.318884 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0618 22:20:28.319008 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:20:28.319611 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:20:28.319616 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:20:28.319639 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:20:28.319787 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:28.319794 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:28.321388 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:28.321391 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:28.322721 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:28.322732 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:28.323715 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:28.323718 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:28.323823 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:28.323839 (+ 16us) client_negotiation.cc:769] Sending connection context
0618 22:20:28.323914 (+ 75us) client_negotiation.cc:240] Negotiation successful
0618 22:20:28.324056 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":810,"mutex_wait_us":54,"thread_start_us":76,"threads_started":1}
I20260618 22:20:28.324224 14648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:28.318255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:43361 (local address 127.0.0.1:41674)
0618 22:20:28.319283 (+ 1028us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:28.319296 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:20:28.319366 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:20:28.319801 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:20:28.319804 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:20:28.319823 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:20:28.319928 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:28.319938 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:28.321145 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:28.321153 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:28.322527 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:28.322537 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:28.323628 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:28.323632 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:28.323751 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:28.323809 (+ 58us) client_negotiation.cc:769] Sending connection context
0618 22:20:28.323914 (+ 105us) client_negotiation.cc:240] Negotiation successful
0618 22:20:28.324056 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":970,"mutex_wait_us":81,"thread_start_us":107,"threads_started":1}
I20260618 22:20:28.324942 14653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:28.318660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45320 (local address 127.0.98.66:46349)
0618 22:20:28.319789 (+ 1129us) server_negotiation.cc:206] Beginning negotiation
0618 22:20:28.319801 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0618 22:20:28.319821 (+ 20us) server_negotiation.cc:407] Connection header received
0618 22:20:28.319892 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:20:28.319899 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:20:28.320096 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:20:28.320241 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:20:28.320483 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:28.321548 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:28.322956 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:28.323847 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:28.323973 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:28.324097 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0618 22:20:28.324723 (+ 626us) server_negotiation.cc:299] Negotiation successful
0618 22:20:28.324805 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":133,"threads_started":1}
I20260618 22:20:28.324633 14649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:28.317974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:46349 (local address 127.0.0.1:45320)
0618 22:20:28.319237 (+ 1263us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:28.319250 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:20:28.319346 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:20:28.320246 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:20:28.320250 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:20:28.320269 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:20:28.320353 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:28.320359 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:28.321684 (+ 1325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:28.321687 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:28.322826 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:28.322837 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:28.324163 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:28.324166 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:28.324284 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:28.324319 (+ 35us) client_negotiation.cc:769] Sending connection context
0618 22:20:28.324419 (+ 100us) client_negotiation.cc:240] Negotiation successful
0618 22:20:28.324499 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1202,"thread_start_us":105,"threads_started":1}
I20260618 22:20:28.327491 14303 tablet_service.cc:1467] Processing CreateTablet for tablet 20fae18029b64c0884797870897106ed (DEFAULT_TABLE table=test-table [id=57ebb968c88b40c5ae75b73623c63a01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:20:28.328181 14437 tablet_service.cc:1467] Processing CreateTablet for tablet 20fae18029b64c0884797870897106ed (DEFAULT_TABLE table=test-table [id=57ebb968c88b40c5ae75b73623c63a01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:20:28.328677 14303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20fae18029b64c0884797870897106ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:20:28.329183 14437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20fae18029b64c0884797870897106ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:20:28.324923 14652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:28.318660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41674 (local address 127.0.98.67:43361)
0618 22:20:28.319072 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0618 22:20:28.319084 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0618 22:20:28.319374 (+ 290us) server_negotiation.cc:407] Connection header received
0618 22:20:28.319458 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:20:28.319465 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:20:28.319643 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:20:28.319784 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:20:28.320078 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:28.321006 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:28.322669 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:28.323487 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:28.323611 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:28.323744 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0618 22:20:28.324667 (+ 923us) server_negotiation.cc:299] Negotiation successful
0618 22:20:28.324769 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":120,"threads_started":1}
I20260618 22:20:28.331828 14571 tablet_service.cc:1467] Processing CreateTablet for tablet 20fae18029b64c0884797870897106ed (DEFAULT_TABLE table=test-table [id=57ebb968c88b40c5ae75b73623c63a01]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:20:28.332590 14571 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20fae18029b64c0884797870897106ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:20:28.336377 14654 tablet_bootstrap.cc:492] T 20fae18029b64c0884797870897106ed P a2c7bdc59554485ba027d28fee230475: Bootstrap starting.
I20260618 22:20:28.337621 14655 tablet_bootstrap.cc:492] T 20fae18029b64c0884797870897106ed P 59d6036ca6d44f6fb120d8c00a3c3665: Bootstrap starting.
I20260618 22:20:28.338977 14654 tablet_bootstrap.cc:654] T 20fae18029b64c0884797870897106ed P a2c7bdc59554485ba027d28fee230475: Neither blocks nor log segments found. Creating new log.
I20260618 22:20:28.339255 14656 tablet_bootstrap.cc:492] T 20fae18029b64c0884797870897106ed P 222f7b84f00d4e38afcb56c5cc3a1455: Bootstrap starting.
I20260618 22:20:28.340081 14654 log.cc:824] T 20fae18029b64c0884797870897106ed P a2c7bdc59554485ba027d28fee230475: Log is configured to *not* fsync() on all Append() calls
I20260618 22:20:28.340286 14655 tablet_bootstrap.cc:654] T 20fae18029b64c0884797870897106ed P 59d6036ca6d44f6fb120d8c00a3c3665: Neither blocks nor log segments found. Creating new log.
I20260618 22:20:28.341178 14655 log.cc:824] T 20fae18029b64c0884797870897106ed P 59d6036ca6d44f6fb120d8c00a3c3665: Log is configured to *not* fsync() on all Append() calls
I20260618 22:20:28.341564 14656 tablet_bootstrap.cc:654] T 20fae18029b64c0884797870897106ed P 222f7b84f00d4e38afcb56c5cc3a1455: Neither blocks nor log segments found. Creating new log.
I20260618 22:20:28.342031 14654 tablet_bootstrap.cc:492] T 20fae18029b64c0884797870897106ed P a2c7bdc59554485ba027d28fee230475: No bootstrap required, opened a new log
I20260618 22:20:28.342218 14654 ts_tablet_manager.cc:1397] T 20fae18029b64c0884797870897106ed P a2c7bdc59554485ba027d28fee230475: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260618 22:20:28.343024 14656 log.cc:824] T 20fae18029b64c0884797870897106ed P 222f7b84f00d4e38afcb56c5cc3a1455: Log is configured to *not* fsync() on all Append() calls
I20260618 22:20:28.343513 14655 tablet_bootstrap.cc:492] T 20fae18029b64c0884797870897106ed P 59d6036ca6d44f6fb120d8c00a3c3665: No bootstrap required, opened a new log
I20260618 22:20:28.343731 14655 ts_tablet_manager.cc:1397] T 20fae18029b64c0884797870897106ed P 59d6036ca6d44f6fb120d8c00a3c3665: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260618 22:20:28.344981 14656 tablet_bootstrap.cc:492] T 20fae18029b64c0884797870897106ed P 222f7b84f00d4e38afcb56c5cc3a1455: No bootstrap required, opened a new log
I20260618 22:20:28.345160 14656 ts_tablet_manager.cc:1397] T 20fae18029b64c0884797870897106ed P 222f7b84f00d4e38afcb56c5cc3a1455: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20260618 22:20:28.345690 14654 raft_consensus.cc:348] T 20fae18029b64c0884797870897106ed P a2c7bdc59554485ba027d28fee230475 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c7bdc59554485ba027d28fee230475" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34547 } } peers { permanent_uuid: "59d6036ca6d44f6fb120d8c00a3c3665" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46349 } } peers { permanent_uuid: "222f7b84f00d4e38afcb56c5cc3a1455" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43361 } }
I20260618 22:20:28.345961 14654 raft_consensus.cc:374] T 20fae18029b64c0884797870897106ed P a2c7bdc59554485ba027d28fee230475 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:20:28.346017 14654 raft_consensus.cc:729] T 20fae18029b64c0884797870897106ed P a2c7bdc59554485ba027d28fee230475 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c7bdc59554485ba027d28fee230475, State: Initialized, Role: FOLLOWER
I20260618 22:20:28.346437 14654 consensus_queue.cc:260] T 20fae18029b64c0884797870897106ed P a2c7bdc59554485ba027d28fee230475 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c7bdc59554485ba027d28fee230475" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34547 } } peers { permanent_uuid: "59d6036ca6d44f6fb120d8c00a3c3665" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46349 } } peers { permanent_uuid: "222f7b84f00d4e38afcb56c5cc3a1455" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43361 } }
I20260618 22:20:28.347064 14369 heartbeater.cc:503] Master 127.0.98.126:38557 was elected leader, sending a full tablet report...
I20260618 22:20:28.347571 14655 raft_consensus.cc:348] T 20fae18029b64c0884797870897106ed P 59d6036ca6d44f6fb120d8c00a3c3665 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c7bdc59554485ba027d28fee230475" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34547 } } peers { permanent_uuid: "59d6036ca6d44f6fb120d8c00a3c3665" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46349 } } peers { permanent_uuid: "222f7b84f00d4e38afcb56c5cc3a1455" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43361 } }
I20260618 22:20:28.347838 14655 raft_consensus.cc:374] T 20fae18029b64c0884797870897106ed P 59d6036ca6d44f6fb120d8c00a3c3665 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:20:28.347891 14655 raft_consensus.cc:729] T 20fae18029b64c0884797870897106ed P 59d6036ca6d44f6fb120d8c00a3c3665 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59d6036ca6d44f6fb120d8c00a3c3665, State: Initialized, Role: FOLLOWER
I20260618 22:20:28.348345 14654 ts_tablet_manager.cc:1428] T 20fae18029b64c0884797870897106ed P a2c7bdc59554485ba027d28fee230475: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260618 22:20:28.348279 14655 consensus_queue.cc:260] T 20fae18029b64c0884797870897106ed P 59d6036ca6d44f6fb120d8c00a3c3665 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c7bdc59554485ba027d28fee230475" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34547 } } peers { permanent_uuid: "59d6036ca6d44f6fb120d8c00a3c3665" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46349 } } peers { permanent_uuid: "222f7b84f00d4e38afcb56c5cc3a1455" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43361 } }
I20260618 22:20:28.348991 14503 heartbeater.cc:503] Master 127.0.98.126:38557 was elected leader, sending a full tablet report...
I20260618 22:20:28.349267 14655 ts_tablet_manager.cc:1428] T 20fae18029b64c0884797870897106ed P 59d6036ca6d44f6fb120d8c00a3c3665: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260618 22:20:28.349606 14656 raft_consensus.cc:348] T 20fae18029b64c0884797870897106ed P 222f7b84f00d4e38afcb56c5cc3a1455 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c7bdc59554485ba027d28fee230475" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34547 } } peers { permanent_uuid: "59d6036ca6d44f6fb120d8c00a3c3665" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46349 } } peers { permanent_uuid: "222f7b84f00d4e38afcb56c5cc3a1455" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43361 } }
I20260618 22:20:28.349918 14656 raft_consensus.cc:374] T 20fae18029b64c0884797870897106ed P 222f7b84f00d4e38afcb56c5cc3a1455 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:20:28.349987 14656 raft_consensus.cc:729] T 20fae18029b64c0884797870897106ed P 222f7b84f00d4e38afcb56c5cc3a1455 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 222f7b84f00d4e38afcb56c5cc3a1455, State: Initialized, Role: FOLLOWER
I20260618 22:20:28.350337 14656 consensus_queue.cc:260] T 20fae18029b64c0884797870897106ed P 222f7b84f00d4e38afcb56c5cc3a1455 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c7bdc59554485ba027d28fee230475" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34547 } } peers { permanent_uuid: "59d6036ca6d44f6fb120d8c00a3c3665" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46349 } } peers { permanent_uuid: "222f7b84f00d4e38afcb56c5cc3a1455" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43361 } }
I20260618 22:20:28.350967 14637 heartbeater.cc:503] Master 127.0.98.126:38557 was elected leader, sending a full tablet report...
I20260618 22:20:28.351251 14656 ts_tablet_manager.cc:1428] T 20fae18029b64c0884797870897106ed P 222f7b84f00d4e38afcb56c5cc3a1455: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
W20260618 22:20:28.363664 14370 tablet.cc:2369] T 20fae18029b64c0884797870897106ed P a2c7bdc59554485ba027d28fee230475: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260618 22:20:28.525552 14638 tablet.cc:2369] T 20fae18029b64c0884797870897106ed P 222f7b84f00d4e38afcb56c5cc3a1455: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260618 22:20:28.575635 14504 tablet.cc:2369] T 20fae18029b64c0884797870897106ed P 59d6036ca6d44f6fb120d8c00a3c3665: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:20:28.678560 14662 raft_consensus.cc:482] T 20fae18029b64c0884797870897106ed P 222f7b84f00d4e38afcb56c5cc3a1455 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:20:28.678787 14662 raft_consensus.cc:504] T 20fae18029b64c0884797870897106ed P 222f7b84f00d4e38afcb56c5cc3a1455 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c7bdc59554485ba027d28fee230475" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34547 } } peers { permanent_uuid: "59d6036ca6d44f6fb120d8c00a3c3665" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46349 } } peers { permanent_uuid: "222f7b84f00d4e38afcb56c5cc3a1455" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43361 } }
I20260618 22:20:28.679960 14662 leader_election.cc:288] T 20fae18029b64c0884797870897106ed P 222f7b84f00d4e38afcb56c5cc3a1455 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2c7bdc59554485ba027d28fee230475 (127.0.98.65:34547), 59d6036ca6d44f6fb120d8c00a3c3665 (127.0.98.66:46349)
I20260618 22:20:28.684912 14653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:28.680367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:33341 (local address 127.0.98.66:46349)
0618 22:20:28.680504 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0618 22:20:28.680509 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:20:28.680536 (+ 27us) server_negotiation.cc:407] Connection header received
0618 22:20:28.680593 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:20:28.680596 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:20:28.680638 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:20:28.680746 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:20:28.681018 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:28.681941 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:28.683536 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:28.684218 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:28.684352 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:28.684418 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0618 22:20:28.684700 (+ 282us) server_negotiation.cc:299] Negotiation successful
0618 22:20:28.684776 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260618 22:20:28.684921 14639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:28.680220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:46349 (local address 127.0.98.67:33341)
0618 22:20:28.680410 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:28.680425 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:20:28.680516 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:20:28.680771 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:20:28.680776 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:20:28.680796 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:20:28.680912 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:28.680916 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:28.682054 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:28.682056 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:28.683388 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:28.683399 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:28.684362 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:28.684366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:28.684535 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:28.684584 (+ 49us) client_negotiation.cc:769] Sending connection context
0618 22:20:28.684665 (+ 81us) client_negotiation.cc:240] Negotiation successful
0618 22:20:28.684759 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":83}
I20260618 22:20:28.684945 14650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:28.680285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:46637 (local address 127.0.98.65:34547)
0618 22:20:28.680412 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0618 22:20:28.680418 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:20:28.680727 (+ 309us) server_negotiation.cc:407] Connection header received
0618 22:20:28.680902 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:20:28.680904 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:20:28.680978 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:20:28.681068 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:20:28.681292 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:28.682107 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:28.683536 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:28.684244 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:28.684363 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:28.684418 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0618 22:20:28.684700 (+ 282us) server_negotiation.cc:299] Negotiation successful
0618 22:20:28.684776 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260618 22:20:28.684943 14663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:28.680235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:34547 (local address 127.0.98.67:46637)
0618 22:20:28.680628 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:28.680639 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 22:20:28.680738 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:20:28.681086 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:20:28.681088 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:20:28.681103 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:20:28.681187 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:28.681191 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:28.682201 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:28.682203 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:28.683387 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:28.683399 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:28.684362 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:28.684366 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:28.684537 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:28.684590 (+ 53us) client_negotiation.cc:769] Sending connection context
0618 22:20:28.684665 (+ 75us) client_negotiation.cc:240] Negotiation successful
0618 22:20:28.684758 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":321,"mutex_wait_us":105,"thread_start_us":115,"threads_started":1}
I20260618 22:20:28.685647 14457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fae18029b64c0884797870897106ed" candidate_uuid: "222f7b84f00d4e38afcb56c5cc3a1455" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59d6036ca6d44f6fb120d8c00a3c3665" is_pre_election: true
I20260618 22:20:28.685647 14323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fae18029b64c0884797870897106ed" candidate_uuid: "222f7b84f00d4e38afcb56c5cc3a1455" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c7bdc59554485ba027d28fee230475" is_pre_election: true
I20260618 22:20:28.685978 14457 raft_consensus.cc:2454] T 20fae18029b64c0884797870897106ed P 59d6036ca6d44f6fb120d8c00a3c3665 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 222f7b84f00d4e38afcb56c5cc3a1455 in term 0.
I20260618 22:20:28.686000 14323 raft_consensus.cc:2454] T 20fae18029b64c0884797870897106ed P a2c7bdc59554485ba027d28fee230475 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 222f7b84f00d4e38afcb56c5cc3a1455 in term 0.
I20260618 22:20:28.686496 14524 leader_election.cc:302] T 20fae18029b64c0884797870897106ed P 222f7b84f00d4e38afcb56c5cc3a1455 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 222f7b84f00d4e38afcb56c5cc3a1455, 59d6036ca6d44f6fb120d8c00a3c3665, a2c7bdc59554485ba027d28fee230475; no voters:
I20260618 22:20:28.686736 14662 raft_consensus.cc:2781] T 20fae18029b64c0884797870897106ed P 222f7b84f00d4e38afcb56c5cc3a1455 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:20:28.686839 14662 raft_consensus.cc:482] T 20fae18029b64c0884797870897106ed P 222f7b84f00d4e38afcb56c5cc3a1455 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:20:28.686904 14662 raft_consensus.cc:3037] T 20fae18029b64c0884797870897106ed P 222f7b84f00d4e38afcb56c5cc3a1455 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:20:28.688114 14662 raft_consensus.cc:504] T 20fae18029b64c0884797870897106ed P 222f7b84f00d4e38afcb56c5cc3a1455 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c7bdc59554485ba027d28fee230475" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34547 } } peers { permanent_uuid: "59d6036ca6d44f6fb120d8c00a3c3665" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46349 } } peers { permanent_uuid: "222f7b84f00d4e38afcb56c5cc3a1455" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43361 } }
I20260618 22:20:28.688524 14662 leader_election.cc:288] T 20fae18029b64c0884797870897106ed P 222f7b84f00d4e38afcb56c5cc3a1455 [CANDIDATE]: Term 1 election: Requested vote from peers a2c7bdc59554485ba027d28fee230475 (127.0.98.65:34547), 59d6036ca6d44f6fb120d8c00a3c3665 (127.0.98.66:46349)
I20260618 22:20:28.688894 14457 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fae18029b64c0884797870897106ed" candidate_uuid: "222f7b84f00d4e38afcb56c5cc3a1455" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "59d6036ca6d44f6fb120d8c00a3c3665"
I20260618 22:20:28.688864 14323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20fae18029b64c0884797870897106ed" candidate_uuid: "222f7b84f00d4e38afcb56c5cc3a1455" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c7bdc59554485ba027d28fee230475"
I20260618 22:20:28.688999 14457 raft_consensus.cc:3037] T 20fae18029b64c0884797870897106ed P 59d6036ca6d44f6fb120d8c00a3c3665 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:20:28.689105 14323 raft_consensus.cc:3037] T 20fae18029b64c0884797870897106ed P a2c7bdc59554485ba027d28fee230475 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:20:28.690116 14457 raft_consensus.cc:2454] T 20fae18029b64c0884797870897106ed P 59d6036ca6d44f6fb120d8c00a3c3665 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 222f7b84f00d4e38afcb56c5cc3a1455 in term 1.
I20260618 22:20:28.690233 14323 raft_consensus.cc:2454] T 20fae18029b64c0884797870897106ed P a2c7bdc59554485ba027d28fee230475 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 222f7b84f00d4e38afcb56c5cc3a1455 in term 1.
I20260618 22:20:28.690392 14524 leader_election.cc:302] T 20fae18029b64c0884797870897106ed P 222f7b84f00d4e38afcb56c5cc3a1455 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 222f7b84f00d4e38afcb56c5cc3a1455, 59d6036ca6d44f6fb120d8c00a3c3665; no voters:
I20260618 22:20:28.690572 14662 raft_consensus.cc:2781] T 20fae18029b64c0884797870897106ed P 222f7b84f00d4e38afcb56c5cc3a1455 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:20:28.690784 14662 raft_consensus.cc:686] T 20fae18029b64c0884797870897106ed P 222f7b84f00d4e38afcb56c5cc3a1455 [term 1 LEADER]: Becoming Leader. State: Replica: 222f7b84f00d4e38afcb56c5cc3a1455, State: Running, Role: LEADER
I20260618 22:20:28.691097 14662 consensus_queue.cc:237] T 20fae18029b64c0884797870897106ed P 222f7b84f00d4e38afcb56c5cc3a1455 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c7bdc59554485ba027d28fee230475" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34547 } } peers { permanent_uuid: "59d6036ca6d44f6fb120d8c00a3c3665" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46349 } } peers { permanent_uuid: "222f7b84f00d4e38afcb56c5cc3a1455" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43361 } }
I20260618 22:20:28.694214 14177 catalog_manager.cc:5495] T 20fae18029b64c0884797870897106ed P 222f7b84f00d4e38afcb56c5cc3a1455 reported cstate change: term changed from 0 to 1, leader changed from <none> to 222f7b84f00d4e38afcb56c5cc3a1455 (127.0.98.67). New cstate: current_term: 1 leader_uuid: "222f7b84f00d4e38afcb56c5cc3a1455" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c7bdc59554485ba027d28fee230475" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "59d6036ca6d44f6fb120d8c00a3c3665" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "222f7b84f00d4e38afcb56c5cc3a1455" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43361 } health_report { overall_health: HEALTHY } } }
I20260618 22:20:28.771121 14652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:28.767484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41680 (local address 127.0.98.67:43361)
0618 22:20:28.767652 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0618 22:20:28.767656 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:20:28.767669 (+ 13us) server_negotiation.cc:407] Connection header received
0618 22:20:28.767749 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:20:28.767751 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:20:28.767802 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:20:28.767915 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:20:28.768190 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:28.768708 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:28.769580 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:28.769802 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:28.769946 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:28.770237 (+ 291us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:20:28.770654 (+ 417us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:20:28.770792 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0618 22:20:28.770880 (+ 88us) server_negotiation.cc:299] Negotiation successful
0618 22:20:28.770936 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260618 22:20:28.776757 14323 raft_consensus.cc:1261] T 20fae18029b64c0884797870897106ed P a2c7bdc59554485ba027d28fee230475 [term 1 FOLLOWER]: Refusing update from remote peer 222f7b84f00d4e38afcb56c5cc3a1455: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:20:28.776757 14457 raft_consensus.cc:1261] T 20fae18029b64c0884797870897106ed P 59d6036ca6d44f6fb120d8c00a3c3665 [term 1 FOLLOWER]: Refusing update from remote peer 222f7b84f00d4e38afcb56c5cc3a1455: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:20:28.777474 14664 consensus_queue.cc:1035] T 20fae18029b64c0884797870897106ed P 222f7b84f00d4e38afcb56c5cc3a1455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c7bdc59554485ba027d28fee230475" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:20:28.777637 14662 consensus_queue.cc:1035] T 20fae18029b64c0884797870897106ed P 222f7b84f00d4e38afcb56c5cc3a1455 [LEADER]: Connected to new peer: Peer: permanent_uuid: "59d6036ca6d44f6fb120d8c00a3c3665" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:20:28.783242 14667 mvcc.cc:205] Tried to move back new op lower bound from 7298339753062649856 to 7298339752721035264. Current Snapshot: MvccSnapshot[applied={T|T < 7298339753062649856}]
I20260618 22:20:28.786278 14669 mvcc.cc:205] Tried to move back new op lower bound from 7298339753062649856 to 7298339752721035264. Current Snapshot: MvccSnapshot[applied={T|T < 7298339753062649856}]
I20260618 22:20:28.792124 14177 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:20:28.792313 14177 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:20:28.794914 14177 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dbe151a423ee4337936f2226d65741da: Sending DeleteTablet for 3 replicas of tablet 20fae18029b64c0884797870897106ed
I20260618 22:20:28.795722 14303 tablet_service.cc:1514] Processing DeleteTablet for tablet 20fae18029b64c0884797870897106ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:20:28 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:55722
I20260618 22:20:28.795786 14571 tablet_service.cc:1514] Processing DeleteTablet for tablet 20fae18029b64c0884797870897106ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:20:28 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:41674
I20260618 22:20:28.796267 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 14240
I20260618 22:20:28.796257 14676 tablet_replica.cc:330] T 20fae18029b64c0884797870897106ed P 222f7b84f00d4e38afcb56c5cc3a1455: stopping tablet replica
I20260618 22:20:28.796322 14437 tablet_service.cc:1514] Processing DeleteTablet for tablet 20fae18029b64c0884797870897106ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:20:28 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:45320
I20260618 22:20:28.796729 14676 raft_consensus.cc:2229] T 20fae18029b64c0884797870897106ed P 222f7b84f00d4e38afcb56c5cc3a1455 [term 1 LEADER]: Raft consensus shutting down.
I20260618 22:20:28.797137 14676 raft_consensus.cc:2258] T 20fae18029b64c0884797870897106ed P 222f7b84f00d4e38afcb56c5cc3a1455 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:20:28.798658 14676 ts_tablet_manager.cc:1905] T 20fae18029b64c0884797870897106ed P 222f7b84f00d4e38afcb56c5cc3a1455: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:20:28.799108 14678 tablet_replica.cc:330] T 20fae18029b64c0884797870897106ed P 59d6036ca6d44f6fb120d8c00a3c3665: stopping tablet replica
I20260618 22:20:28.799376 14678 raft_consensus.cc:2229] T 20fae18029b64c0884797870897106ed P 59d6036ca6d44f6fb120d8c00a3c3665 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:20:28.799686 14678 raft_consensus.cc:2258] T 20fae18029b64c0884797870897106ed P 59d6036ca6d44f6fb120d8c00a3c3665 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:20:28.801188 14678 ts_tablet_manager.cc:1905] T 20fae18029b64c0884797870897106ed P 59d6036ca6d44f6fb120d8c00a3c3665: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:20:28.801481 14676 ts_tablet_manager.cc:1918] T 20fae18029b64c0884797870897106ed P 222f7b84f00d4e38afcb56c5cc3a1455: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:20:28.801585 14676 log.cc:1196] T 20fae18029b64c0884797870897106ed P 222f7b84f00d4e38afcb56c5cc3a1455: Deleting WAL directory at /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-2/wal/wals/20fae18029b64c0884797870897106ed
I20260618 22:20:28.801961 14676 ts_tablet_manager.cc:1939] T 20fae18029b64c0884797870897106ed P 222f7b84f00d4e38afcb56c5cc3a1455: Deleting consensus metadata
I20260618 22:20:28.803404 14163 catalog_manager.cc:4841] TS 222f7b84f00d4e38afcb56c5cc3a1455 (127.0.98.67:43361): tablet 20fae18029b64c0884797870897106ed (table test-table [id=57ebb968c88b40c5ae75b73623c63a01]) successfully deleted
I20260618 22:20:28.804169 14678 ts_tablet_manager.cc:1918] T 20fae18029b64c0884797870897106ed P 59d6036ca6d44f6fb120d8c00a3c3665: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:20:28.804368 14678 log.cc:1196] T 20fae18029b64c0884797870897106ed P 59d6036ca6d44f6fb120d8c00a3c3665: Deleting WAL directory at /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-1/wal/wals/20fae18029b64c0884797870897106ed
I20260618 22:20:28.804694 14678 ts_tablet_manager.cc:1939] T 20fae18029b64c0884797870897106ed P 59d6036ca6d44f6fb120d8c00a3c3665: Deleting consensus metadata
W20260618 22:20:28.805557 14164 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.65:34547 (error 108)
I20260618 22:20:28.805971 14162 catalog_manager.cc:4841] TS 59d6036ca6d44f6fb120d8c00a3c3665 (127.0.98.66:46349): tablet 20fae18029b64c0884797870897106ed (table test-table [id=57ebb968c88b40c5ae75b73623c63a01]) successfully deleted
I20260618 22:20:28.806367 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 14374
I20260618 22:20:28.806680 14649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:28.806408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:34547 (local address 127.0.0.1:55726)
0618 22:20:28.806519 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:28.806590 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.65:34547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260618 22:20:28.807580 14164 catalog_manager.cc:4584] TS a2c7bdc59554485ba027d28fee230475 (127.0.98.65:34547): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 20fae18029b64c0884797870897106ed: Network error: Client connection negotiation failed: client connection to 127.0.98.65:34547: connect: Connection refused (error 111)
I20260618 22:20:28.812434 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 14508
I20260618 22:20:28.817094 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 14146
I20260618 22:20:28.823099 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:38557
--webserver_interface=127.0.98.126
--webserver_port=44585
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:43207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:38557
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:20:28.932480 14680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:20:28.932711 14680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:20:28.932749 14680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:20:28.936666 14680 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:20:28.936715 14680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:20:28.936734 14680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:20:28.936750 14680 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:20:28.936765 14680 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:20:28.941802 14680 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:43207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:38557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:38557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=44585
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:20:28.942700 14680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:20:28.943584 14680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:20:28.955832 14687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:20:28.955931 14689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:20:28.955837 14686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:20:28.956370 14680 server_base.cc:1034] running on GCE node
I20260618 22:20:28.957006 14680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:20:28.958242 14680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:20:28.959432 14680 hybrid_clock.cc:648] HybridClock initialized: now 1781821228959416 us; error 40 us; skew 500 ppm
Jun 18 22:20:28 dist-test-slave-g6r4 krb5kdc[14133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821228, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:20:28.962175 14680 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260618 22:20:28.963130 14680 webserver.cc:460] Webserver started at http://127.0.98.126:44585/ using document root <none> and password file <none>
I20260618 22:20:28.963726 14680 fs_manager.cc:359] Metadata directory not provided
I20260618 22:20:28.963852 14680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:20:28.967845 14680 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260618 22:20:28.970060 14696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:20:28.971275 14680 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260618 22:20:28.971436 14680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/master-0/wal
uuid: "dbe151a423ee4337936f2226d65741da"
format_stamp: "Formatted at 2026-06-18 22:20:27 on dist-test-slave-g6r4"
server_key: "31a3e970244938294a9a6f0a346c5376"
server_key_iv: "ae14f4c650bcd0940dbaee4523a3337b"
server_key_version: "encryptionkey@0"
I20260618 22:20:28.972028 14680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:20:28.994599 14680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:20:28.997296 14680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:20:28.997496 14680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:20:29.005236 14680 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:38557
I20260618 22:20:29.005604 14748 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:38557 every 8 connection(s)
I20260618 22:20:29.006337 14680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
I20260618 22:20:29.009305 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 14680
I20260618 22:20:29.009764 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:34547
--local_ip_for_outbound_sockets=127.0.98.65
--tserver_master_addrs=127.0.98.126:38557
--webserver_port=45317
--webserver_interface=127.0.98.65
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:43207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 22:20:29.011061 14749 sys_catalog.cc:263] Verifying existing consensus state
I20260618 22:20:29.012600 14749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbe151a423ee4337936f2226d65741da: Bootstrap starting.
I20260618 22:20:29.029003 14749 log.cc:824] T 00000000000000000000000000000000 P dbe151a423ee4337936f2226d65741da: Log is configured to *not* fsync() on all Append() calls
I20260618 22:20:29.037165 14749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbe151a423ee4337936f2226d65741da: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260618 22:20:29.037539 14749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbe151a423ee4337936f2226d65741da: Bootstrap complete.
I20260618 22:20:29.041505 14749 raft_consensus.cc:348] T 00000000000000000000000000000000 P dbe151a423ee4337936f2226d65741da [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbe151a423ee4337936f2226d65741da" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38557 } }
I20260618 22:20:29.041893 14749 raft_consensus.cc:729] T 00000000000000000000000000000000 P dbe151a423ee4337936f2226d65741da [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbe151a423ee4337936f2226d65741da, State: Initialized, Role: FOLLOWER
I20260618 22:20:29.042421 14749 consensus_queue.cc:260] T 00000000000000000000000000000000 P dbe151a423ee4337936f2226d65741da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbe151a423ee4337936f2226d65741da" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38557 } }
I20260618 22:20:29.042573 14749 raft_consensus.cc:388] T 00000000000000000000000000000000 P dbe151a423ee4337936f2226d65741da [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:20:29.042659 14749 raft_consensus.cc:482] T 00000000000000000000000000000000 P dbe151a423ee4337936f2226d65741da [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:20:29.042774 14749 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dbe151a423ee4337936f2226d65741da [term 1 FOLLOWER]: Advancing to term 2
I20260618 22:20:29.044615 14749 raft_consensus.cc:504] T 00000000000000000000000000000000 P dbe151a423ee4337936f2226d65741da [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbe151a423ee4337936f2226d65741da" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38557 } }
I20260618 22:20:29.044930 14749 leader_election.cc:302] T 00000000000000000000000000000000 P dbe151a423ee4337936f2226d65741da [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dbe151a423ee4337936f2226d65741da; no voters:
I20260618 22:20:29.045251 14749 leader_election.cc:288] T 00000000000000000000000000000000 P dbe151a423ee4337936f2226d65741da [CANDIDATE]: Term 2 election: Requested vote from peers
I20260618 22:20:29.045334 14753 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dbe151a423ee4337936f2226d65741da [term 2 FOLLOWER]: Leader election won for term 2
I20260618 22:20:29.045557 14753 raft_consensus.cc:686] T 00000000000000000000000000000000 P dbe151a423ee4337936f2226d65741da [term 2 LEADER]: Becoming Leader. State: Replica: dbe151a423ee4337936f2226d65741da, State: Running, Role: LEADER
I20260618 22:20:29.045832 14753 consensus_queue.cc:237] T 00000000000000000000000000000000 P dbe151a423ee4337936f2226d65741da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbe151a423ee4337936f2226d65741da" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38557 } }
I20260618 22:20:29.046425 14749 sys_catalog.cc:564] T 00000000000000000000000000000000 P dbe151a423ee4337936f2226d65741da [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:20:29.046751 14753 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbe151a423ee4337936f2226d65741da [sys.catalog]: SysCatalogTable state changed. Reason: New leader dbe151a423ee4337936f2226d65741da. Latest consensus state: current_term: 2 leader_uuid: "dbe151a423ee4337936f2226d65741da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbe151a423ee4337936f2226d65741da" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38557 } } }
I20260618 22:20:29.046865 14753 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbe151a423ee4337936f2226d65741da [sys.catalog]: This master's current role is: LEADER
I20260618 22:20:29.047273 14754 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbe151a423ee4337936f2226d65741da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "dbe151a423ee4337936f2226d65741da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbe151a423ee4337936f2226d65741da" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38557 } } }
I20260618 22:20:29.047454 14754 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbe151a423ee4337936f2226d65741da [sys.catalog]: This master's current role is: LEADER
I20260618 22:20:29.048517 14757 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:20:29.052282 14757 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:20:29.052778 14757 catalog_manager.cc:1259] Loaded cluster ID: 42893856c82e4dcbb1a581319025cfb1
I20260618 22:20:29.052850 14757 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:20:29.053995 14757 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260618 22:20:29.056675 14757 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260618 22:20:29.063655 14757 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:20:29.064347 14757 catalog_manager.cc:5879] T 00000000000000000000000000000000 P dbe151a423ee4337936f2226d65741da: Loaded TSK: 0
I20260618 22:20:29.065233 14757 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260618 22:20:29.158756 14751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:20:29.159145 14751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:20:29.159215 14751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:20:29.165694 14751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:20:29.165774 14751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:20:29.165870 14751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260618 22:20:29.174171 14751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:43207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:34547
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=45317
--tserver_master_addrs=127.0.98.126:38557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:20:29.175505 14751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:20:29.176816 14751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:20:29.187752 14779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:20:29.187736 14776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:20:29.187736 14777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:20:29.188241 14751 server_base.cc:1034] running on GCE node
I20260618 22:20:29.188695 14751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:20:29.189456 14751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:20:29.190663 14751 hybrid_clock.cc:648] HybridClock initialized: now 1781821229190639 us; error 33 us; skew 500 ppm
Jun 18 22:20:29 dist-test-slave-g6r4 krb5kdc[14133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821229, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:20:29.194116 14751 init.cc:377] Logged in from keytab as kudu/127.0.98.65@KRBTEST.COM (short username kudu)
I20260618 22:20:29.195204 14751 webserver.cc:460] Webserver started at http://127.0.98.65:45317/ using document root <none> and password file <none>
I20260618 22:20:29.195909 14751 fs_manager.cc:359] Metadata directory not provided
I20260618 22:20:29.196022 14751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:20:29.201064 14751 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260618 22:20:29.203770 14786 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:20:29.205322 14751 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260618 22:20:29.205448 14751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-0/wal
uuid: "a2c7bdc59554485ba027d28fee230475"
format_stamp: "Formatted at 2026-06-18 22:20:27 on dist-test-slave-g6r4"
server_key: "9a934605ba3da4a73d2275f4cb40e86e"
server_key_iv: "3d1359fb93e157e37f591124f86586f5"
server_key_version: "encryptionkey@0"
I20260618 22:20:29.206204 14751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:20:29.257643 14751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:20:29.262233 14751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:20:29.262437 14751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:20:29.263103 14751 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:20:29.264443 14793 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260618 22:20:29.269804 14751 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260618 22:20:29.269882 14751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260618 22:20:29.269974 14751 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260618 22:20:29.272043 14751 ts_tablet_manager.cc:610] Registered 1 tablets
I20260618 22:20:29.272100 14751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260618 22:20:29.272393 14793 tablet_bootstrap.cc:492] T 20fae18029b64c0884797870897106ed P a2c7bdc59554485ba027d28fee230475: Bootstrap starting.
I20260618 22:20:29.284555 14751 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:34547
I20260618 22:20:29.285840 14751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
I20260618 22:20:29.287112 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 14751
I20260618 22:20:29.287608 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:46349
--local_ip_for_outbound_sockets=127.0.98.66
--tserver_master_addrs=127.0.98.126:38557
--webserver_port=33247
--webserver_interface=127.0.98.66
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:43207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 22:20:29.294085 14900 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:34547 every 8 connection(s)
I20260618 22:20:29.295640 14793 log.cc:824] T 20fae18029b64c0884797870897106ed P a2c7bdc59554485ba027d28fee230475: Log is configured to *not* fsync() on all Append() calls
I20260618 22:20:29.306367 14793 tablet_bootstrap.cc:492] T 20fae18029b64c0884797870897106ed P a2c7bdc59554485ba027d28fee230475: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260618 22:20:29.306757 14793 tablet_bootstrap.cc:492] T 20fae18029b64c0884797870897106ed P a2c7bdc59554485ba027d28fee230475: Bootstrap complete.
Jun 18 22:20:29 dist-test-slave-g6r4 krb5kdc[14133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821229, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:20:29.308163 14793 ts_tablet_manager.cc:1397] T 20fae18029b64c0884797870897106ed P a2c7bdc59554485ba027d28fee230475: Time spent bootstrapping tablet: real 0.036s user 0.026s sys 0.004s
I20260618 22:20:29.311189 14793 raft_consensus.cc:348] T 20fae18029b64c0884797870897106ed P a2c7bdc59554485ba027d28fee230475 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c7bdc59554485ba027d28fee230475" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34547 } } peers { permanent_uuid: "59d6036ca6d44f6fb120d8c00a3c3665" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46349 } } peers { permanent_uuid: "222f7b84f00d4e38afcb56c5cc3a1455" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43361 } }
I20260618 22:20:29.311710 14793 raft_consensus.cc:729] T 20fae18029b64c0884797870897106ed P a2c7bdc59554485ba027d28fee230475 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c7bdc59554485ba027d28fee230475, State: Initialized, Role: FOLLOWER
I20260618 22:20:29.312321 14793 consensus_queue.cc:260] T 20fae18029b64c0884797870897106ed P a2c7bdc59554485ba027d28fee230475 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2c7bdc59554485ba027d28fee230475" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34547 } } peers { permanent_uuid: "59d6036ca6d44f6fb120d8c00a3c3665" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46349 } } peers { permanent_uuid: "222f7b84f00d4e38afcb56c5cc3a1455" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43361 } }
I20260618 22:20:29.313344 14793 ts_tablet_manager.cc:1428] T 20fae18029b64c0884797870897106ed P a2c7bdc59554485ba027d28fee230475: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260618 22:20:29.314971 14904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:29.288299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38557 (local address 127.0.98.65:43159)
0618 22:20:29.294750 (+ 6451us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:29.294829 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0618 22:20:29.296197 (+ 1368us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:20:29.298422 (+ 2225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:20:29.298429 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:20:29.298792 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:20:29.299242 (+ 450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:29.299255 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:29.301445 (+ 2190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:29.301450 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:29.303155 (+ 1705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:29.303172 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:29.304530 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:29.304535 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:29.304742 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:29.305404 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:20:29.305425 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:20:29.307668 (+ 2243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:20:29.310687 (+ 3019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:20:29.310695 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:20:29.310710 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:20:29.311115 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:20:29.311662 (+ 547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:20:29.311668 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:20:29.311672 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:20:29.311865 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:20:29.312873 (+ 1008us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:20:29.312880 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:20:29.313210 (+ 330us) client_negotiation.cc:769] Sending connection context
0618 22:20:29.313442 (+ 232us) client_negotiation.cc:240] Negotiation successful
0618 22:20:29.313830 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6171,"thread_start_us":149,"threads_started":1}
I20260618 22:20:29.315217 14903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:29.288258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:43159 (local address 127.0.98.126:38557)
0618 22:20:29.296759 (+ 8501us) server_negotiation.cc:206] Beginning negotiation
0618 22:20:29.296795 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0618 22:20:29.296840 (+ 45us) server_negotiation.cc:407] Connection header received
0618 22:20:29.297158 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:20:29.297177 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:20:29.297629 (+ 452us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:20:29.298045 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:20:29.299427 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:29.301229 (+ 1802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:29.303331 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:29.304120 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:29.304321 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:29.307921 (+ 3600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:20:29.307932 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:20:29.307950 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:20:29.307989 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:20:29.310519 (+ 2530us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:29.311312 (+ 793us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:29.311321 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:29.311328 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:29.311504 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:29.312021 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:29.312027 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:29.312032 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:29.312664 (+ 632us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:20:29.312816 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0618 22:20:29.314139 (+ 1323us) server_negotiation.cc:299] Negotiation successful
0618 22:20:29.314431 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8323,"thread_start_us":127,"threads_started":1}
I20260618 22:20:29.317605 14901 heartbeater.cc:347] Connected to a master server at 127.0.98.126:38557
I20260618 22:20:29.317835 14901 heartbeater.cc:464] Registering TS with master...
I20260618 22:20:29.318317 14901 heartbeater.cc:511] Master 127.0.98.126:38557 requested a full tablet report, sending...
I20260618 22:20:29.320962 14713 ts_manager.cc:194] Registered new tserver with Master: a2c7bdc59554485ba027d28fee230475 (127.0.98.65:34547)
I20260618 22:20:29.325541 14713 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.65@KRBTEST.COM'} at 127.0.98.65:43159
I20260618 22:20:29.328228 14901 heartbeater.cc:503] Master 127.0.98.126:38557 was elected leader, sending a full tablet report...
Jun 18 22:20:29 dist-test-slave-g6r4 krb5kdc[14133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821228, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for kudu/127.0.98.65@KRBTEST.COM
I20260618 22:20:29.336371 14909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:29.325445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55734 (local address 127.0.98.65:34547)
0618 22:20:29.325726 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0618 22:20:29.325737 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0618 22:20:29.325760 (+ 23us) server_negotiation.cc:407] Connection header received
0618 22:20:29.325841 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:20:29.325846 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:20:29.326069 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:20:29.326207 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:20:29.326901 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:29.328023 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:29.329299 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:29.329787 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:29.330323 (+ 536us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:29.332839 (+ 2516us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:20:29.332851 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:20:29.332864 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:20:29.332891 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:20:29.334561 (+ 1670us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:29.335076 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:29.335080 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:29.335082 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:29.335158 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:29.335438 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:29.335440 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:29.335441 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:29.335658 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:20:29.335762 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0618 22:20:29.336110 (+ 348us) server_negotiation.cc:299] Negotiation successful
0618 22:20:29.336241 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":105,"threads_started":1}
I20260618 22:20:29.336372 14908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:29.324759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:34547 (local address 127.0.0.1:55734)
0618 22:20:29.325139 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:29.325183 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0618 22:20:29.325299 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:20:29.326245 (+ 946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:20:29.326251 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:20:29.326652 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:20:29.326768 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:29.326776 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:29.328143 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:29.328148 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:29.329141 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:29.329152 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:29.329953 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:29.329957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:29.330075 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:29.330717 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:20:29.330732 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:20:29.332653 (+ 1921us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:20:29.334672 (+ 2019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:20:29.334681 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:20:29.334686 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:20:29.334959 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:20:29.335248 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:20:29.335251 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:20:29.335253 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:20:29.335345 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:20:29.335768 (+ 423us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:20:29.335773 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:20:29.335887 (+ 114us) client_negotiation.cc:769] Sending connection context
0618 22:20:29.336036 (+ 149us) client_negotiation.cc:240] Negotiation successful
0618 22:20:29.336211 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":112,"threads_started":1}
I20260618 22:20:29.337385 14817 tablet_service.cc:1514] Processing DeleteTablet for tablet 20fae18029b64c0884797870897106ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:20:28 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:55734
I20260618 22:20:29.338266 14910 tablet_replica.cc:330] stopping tablet replica
I20260618 22:20:29.338555 14910 raft_consensus.cc:2229] T 20fae18029b64c0884797870897106ed P a2c7bdc59554485ba027d28fee230475 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:20:29.338814 14910 raft_consensus.cc:2258] T 20fae18029b64c0884797870897106ed P a2c7bdc59554485ba027d28fee230475 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:20:29.339813 14910 ts_tablet_manager.cc:1905] T 20fae18029b64c0884797870897106ed P a2c7bdc59554485ba027d28fee230475: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:20:29.343358 14910 ts_tablet_manager.cc:1918] T 20fae18029b64c0884797870897106ed P a2c7bdc59554485ba027d28fee230475: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:20:29.343474 14910 log.cc:1196] T 20fae18029b64c0884797870897106ed P a2c7bdc59554485ba027d28fee230475: Deleting WAL directory at /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-0/wal/wals/20fae18029b64c0884797870897106ed
I20260618 22:20:29.343809 14910 ts_tablet_manager.cc:1939] T 20fae18029b64c0884797870897106ed P a2c7bdc59554485ba027d28fee230475: Deleting consensus metadata
I20260618 22:20:29.345345 14699 catalog_manager.cc:4841] TS a2c7bdc59554485ba027d28fee230475 (127.0.98.65:34547): tablet 20fae18029b64c0884797870897106ed (table test-table [id=57ebb968c88b40c5ae75b73623c63a01]) successfully deleted
W20260618 22:20:29.409356 14905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:20:29.409596 14905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:20:29.409639 14905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:20:29.413743 14905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:20:29.413831 14905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:20:29.413959 14905 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260618 22:20:29.418934 14905 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:43207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:46349
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=33247
--tserver_master_addrs=127.0.98.126:38557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14905
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:20:29.419819 14905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:20:29.420702 14905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:20:29.430773 14916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:20:29.430781 14915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:20:29.430769 14918 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:20:29.430999 14905 server_base.cc:1034] running on GCE node
I20260618 22:20:29.431384 14905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:20:29.431988 14905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:20:29.433174 14905 hybrid_clock.cc:648] HybridClock initialized: now 1781821229433161 us; error 34 us; skew 500 ppm
Jun 18 22:20:29 dist-test-slave-g6r4 krb5kdc[14133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821229, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:20:29.435817 14905 init.cc:377] Logged in from keytab as kudu/127.0.98.66@KRBTEST.COM (short username kudu)
I20260618 22:20:29.436612 14905 webserver.cc:460] Webserver started at http://127.0.98.66:33247/ using document root <none> and password file <none>
I20260618 22:20:29.437146 14905 fs_manager.cc:359] Metadata directory not provided
I20260618 22:20:29.437268 14905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:20:29.441370 14905 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260618 22:20:29.443310 14925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:20:29.444579 14905 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260618 22:20:29.444705 14905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-1/wal
uuid: "59d6036ca6d44f6fb120d8c00a3c3665"
format_stamp: "Formatted at 2026-06-18 22:20:28 on dist-test-slave-g6r4"
server_key: "7cf93c9fd250cd36e81ba505b89d34a6"
server_key_iv: "7564773e47f86984d2b4d5a87d2729ed"
server_key_version: "encryptionkey@0"
I20260618 22:20:29.445225 14905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:20:29.494336 14905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:20:29.497282 14905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:20:29.497431 14905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:20:29.497973 14905 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:20:29.498720 14905 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:20:29.498762 14905 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:20:29.498800 14905 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:20:29.498821 14905 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:20:29.507402 14905 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:46349
I20260618 22:20:29.507440 15038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:46349 every 8 connection(s)
I20260618 22:20:29.508328 14905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
I20260618 22:20:29.515331 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 14905
I20260618 22:20:29.515729 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:43361
--local_ip_for_outbound_sockets=127.0.98.67
--tserver_master_addrs=127.0.98.126:38557
--webserver_port=37883
--webserver_interface=127.0.98.67
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:43207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 18 22:20:29 dist-test-slave-g6r4 krb5kdc[14133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821229, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:20:29.521665 14903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:29.510024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:50515 (local address 127.0.98.126:38557)
0618 22:20:29.510170 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0618 22:20:29.510175 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:20:29.510997 (+ 822us) server_negotiation.cc:407] Connection header received
0618 22:20:29.512295 (+ 1298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:20:29.512298 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:20:29.512342 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:20:29.512415 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:20:29.513565 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:29.514103 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:29.515168 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:29.515400 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:29.515495 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:29.517869 (+ 2374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:20:29.517875 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:20:29.517879 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:20:29.517936 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:20:29.519611 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:29.520076 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:29.520083 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:29.520086 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:29.520179 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:29.520492 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:29.520496 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:29.520501 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:29.520743 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:20:29.520856 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0618 22:20:29.521272 (+ 416us) server_negotiation.cc:299] Negotiation successful
0618 22:20:29.521438 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260618 22:20:29.522192 15041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:29.510433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38557 (local address 127.0.98.66:50515)
0618 22:20:29.510842 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:29.510882 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0618 22:20:29.512112 (+ 1230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:20:29.512571 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:20:29.512578 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:20:29.512984 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:20:29.513437 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:29.513449 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:29.514204 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:29.514207 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:29.515039 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:29.515047 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:29.515544 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:29.515548 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:29.515715 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:29.516300 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:20:29.516315 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:20:29.517685 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:20:29.519728 (+ 2043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:20:29.519732 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:20:29.519742 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:20:29.519975 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:20:29.520280 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:20:29.520282 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:20:29.520284 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:20:29.520403 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:20:29.520848 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:20:29.520853 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:20:29.521048 (+ 195us) client_negotiation.cc:769] Sending connection context
0618 22:20:29.521188 (+ 140us) client_negotiation.cc:240] Negotiation successful
0618 22:20:29.521401 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":104,"threads_started":1}
I20260618 22:20:29.523288 15039 heartbeater.cc:347] Connected to a master server at 127.0.98.126:38557
I20260618 22:20:29.523516 15039 heartbeater.cc:464] Registering TS with master...
I20260618 22:20:29.523959 15039 heartbeater.cc:511] Master 127.0.98.126:38557 requested a full tablet report, sending...
I20260618 22:20:29.524962 14713 ts_manager.cc:194] Registered new tserver with Master: 59d6036ca6d44f6fb120d8c00a3c3665 (127.0.98.66:46349)
I20260618 22:20:29.525504 14713 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.66@KRBTEST.COM'} at 127.0.98.66:50515
W20260618 22:20:29.622558 15042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:20:29.622820 15042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:20:29.622890 15042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:20:29.626842 15042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:20:29.626917 15042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:20:29.627009 15042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260618 22:20:29.632086 15042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:43207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:43361
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=37883
--tserver_master_addrs=127.0.98.126:38557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:20:29.632978 15042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:20:29.633891 15042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:20:29.643208 15050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:20:29.643191 15047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:20:29.643189 15048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:20:29.643708 15042 server_base.cc:1034] running on GCE node
I20260618 22:20:29.644052 15042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:20:29.644654 15042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:20:29.645809 15042 hybrid_clock.cc:648] HybridClock initialized: now 1781821229645752 us; error 68 us; skew 500 ppm
Jun 18 22:20:29 dist-test-slave-g6r4 krb5kdc[14133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821229, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:20:29.648571 15042 init.cc:377] Logged in from keytab as kudu/127.0.98.67@KRBTEST.COM (short username kudu)
I20260618 22:20:29.649509 15042 webserver.cc:460] Webserver started at http://127.0.98.67:37883/ using document root <none> and password file <none>
I20260618 22:20:29.650110 15042 fs_manager.cc:359] Metadata directory not provided
I20260618 22:20:29.650206 15042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:20:29.653867 15042 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:20:29.655822 15057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:20:29.656939 15042 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:20:29.657074 15042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-2/wal
uuid: "222f7b84f00d4e38afcb56c5cc3a1455"
format_stamp: "Formatted at 2026-06-18 22:20:28 on dist-test-slave-g6r4"
server_key: "8b92e37e1fdb89f9145107e6724a1e83"
server_key_iv: "d1ba7ee082abf672580920f1705806b0"
server_key_version: "encryptionkey@0"
I20260618 22:20:29.657611 15042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:20:29.678552 15042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:20:29.681180 15042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:20:29.681376 15042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:20:29.681977 15042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:20:29.682754 15042 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:20:29.682824 15042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:20:29.682891 15042 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:20:29.682957 15042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:20:29.691730 15042 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:43361
I20260618 22:20:29.691764 15170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:43361 every 8 connection(s)
I20260618 22:20:29.692675 15042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
Jun 18 22:20:29 dist-test-slave-g6r4 krb5kdc[14133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821229, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:20:29.702129 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 15042
I20260618 22:20:29.705453 14903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:29.694291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:54821 (local address 127.0.98.126:38557)
0618 22:20:29.694456 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0618 22:20:29.694461 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:20:29.695262 (+ 801us) server_negotiation.cc:407] Connection header received
0618 22:20:29.696524 (+ 1262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:20:29.696527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:20:29.696571 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:20:29.696647 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:20:29.697681 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:29.698258 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:29.699301 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:29.699505 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:29.699579 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:29.701800 (+ 2221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:20:29.701802 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:20:29.701805 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:20:29.701826 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:20:29.703531 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:29.704020 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:29.704023 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:29.704025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:29.704107 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:29.704423 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:29.704427 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:29.704429 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:29.704620 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:20:29.704722 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0618 22:20:29.705188 (+ 466us) server_negotiation.cc:299] Negotiation successful
0618 22:20:29.705291 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260618 22:20:29.706341 15173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:29.694680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38557 (local address 127.0.98.67:54821)
0618 22:20:29.695128 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:29.695168 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0618 22:20:29.696344 (+ 1176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:20:29.696795 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:20:29.696801 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:20:29.697119 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:20:29.697561 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:29.697573 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:29.698361 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:29.698363 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:29.699201 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:29.699209 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:29.699621 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:29.699624 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:29.699759 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:29.700326 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:20:29.700341 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:20:29.701656 (+ 1315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:20:29.703644 (+ 1988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:20:29.703649 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:20:29.703661 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:20:29.703902 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:20:29.704195 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:20:29.704199 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:20:29.704201 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:20:29.704329 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:20:29.704720 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:20:29.704726 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:20:29.704956 (+ 230us) client_negotiation.cc:769] Sending connection context
0618 22:20:29.705132 (+ 176us) client_negotiation.cc:240] Negotiation successful
0618 22:20:29.705412 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":111,"threads_started":1}
I20260618 22:20:29.707561 15171 heartbeater.cc:347] Connected to a master server at 127.0.98.126:38557
I20260618 22:20:29.707813 15171 heartbeater.cc:464] Registering TS with master...
I20260618 22:20:29.708359 15171 heartbeater.cc:511] Master 127.0.98.126:38557 requested a full tablet report, sending...
I20260618 22:20:29.709530 14713 ts_manager.cc:194] Registered new tserver with Master: 222f7b84f00d4e38afcb56c5cc3a1455 (127.0.98.67:43361)
I20260618 22:20:29.710316 14713 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.67@KRBTEST.COM'} at 127.0.98.67:54821
I20260618 22:20:29.711076 15174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:29.702786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45146 (local address 127.0.98.126:38557)
0618 22:20:29.703014 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0618 22:20:29.703017 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:20:29.703280 (+ 263us) server_negotiation.cc:407] Connection header received
0618 22:20:29.703431 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:20:29.703434 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:20:29.703477 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:20:29.703579 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:20:29.704275 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:29.704769 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:29.705782 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:29.706136 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:29.706237 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:29.707843 (+ 1606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:20:29.707848 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:20:29.707851 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:20:29.707874 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:20:29.709408 (+ 1534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:29.709846 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:29.709848 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:29.709849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:29.709973 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:29.710232 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:29.710233 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:29.710235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:29.710399 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:20:29.710587 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0618 22:20:29.710762 (+ 175us) server_negotiation.cc:299] Negotiation successful
0618 22:20:29.710872 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":80,"threads_started":1}
I20260618 22:20:29.712771 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 22:20:29.716662 15174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:29.713329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45162 (local address 127.0.98.126:38557)
0618 22:20:29.713434 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0618 22:20:29.713437 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:20:29.713628 (+ 191us) server_negotiation.cc:407] Connection header received
0618 22:20:29.713739 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:20:29.713741 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:20:29.713819 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:20:29.713893 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:20:29.714210 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:29.714670 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:29.715533 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:29.715740 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:29.715813 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:29.716079 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:20:29.716391 (+ 312us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:20:29.716474 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0618 22:20:29.716536 (+ 62us) server_negotiation.cc:299] Negotiation successful
0618 22:20:29.716579 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260618 22:20:29.717208 14713 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:45162:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 22:20:29.720322 14713 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:20:29.727380 14817 tablet_service.cc:1467] Processing CreateTablet for tablet b1193469535d4d2bb5bdaa63c4303273 (DEFAULT_TABLE table=test-table [id=9dc6b3402f324d27b92514d7b9957d7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:20:29.727990 14817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1193469535d4d2bb5bdaa63c4303273. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:20:29.730768 14793 tablet_bootstrap.cc:492] T b1193469535d4d2bb5bdaa63c4303273 P a2c7bdc59554485ba027d28fee230475: Bootstrap starting.
I20260618 22:20:29.731619 14908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:29.726370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:46349 (local address 127.0.0.1:45330)
0618 22:20:29.726539 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:29.726554 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:20:29.726634 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:20:29.727383 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:20:29.727385 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:20:29.727399 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:20:29.727494 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:29.727500 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:29.728857 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:29.728860 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:29.730086 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:29.730096 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:29.731157 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:29.731161 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:29.731261 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:29.731303 (+ 42us) client_negotiation.cc:769] Sending connection context
0618 22:20:29.731392 (+ 89us) client_negotiation.cc:240] Negotiation successful
0618 22:20:29.731485 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"mutex_wait_us":27}
I20260618 22:20:29.731678 14793 tablet_bootstrap.cc:654] T b1193469535d4d2bb5bdaa63c4303273 P a2c7bdc59554485ba027d28fee230475: Neither blocks nor log segments found. Creating new log.
I20260618 22:20:29.731722 15180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:29.726526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45330 (local address 127.0.98.66:46349)
0618 22:20:29.726861 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0618 22:20:29.726873 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0618 22:20:29.726897 (+ 24us) server_negotiation.cc:407] Connection header received
0618 22:20:29.726974 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:20:29.726982 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:20:29.727214 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:20:29.727412 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:20:29.727588 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:29.728719 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:29.730214 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:29.731033 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:29.731143 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:29.731286 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0618 22:20:29.731423 (+ 137us) server_negotiation.cc:299] Negotiation successful
0618 22:20:29.731529 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":137,"threads_started":1}
I20260618 22:20:29.732329 15181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:29.726792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:43361 (local address 127.0.0.1:41694)
0618 22:20:29.727157 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:29.727170 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:20:29.727252 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:20:29.727898 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:20:29.727902 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:20:29.727920 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:20:29.728008 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:29.728018 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:29.729241 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:29.729247 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:29.730389 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:29.730400 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:29.731926 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:29.731929 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:29.732025 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:29.732047 (+ 22us) client_negotiation.cc:769] Sending connection context
0618 22:20:29.732121 (+ 74us) client_negotiation.cc:240] Negotiation successful
0618 22:20:29.732191 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"mutex_wait_us":1,"thread_start_us":78,"threads_started":1}
I20260618 22:20:29.732791 15182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:29.727100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41694 (local address 127.0.98.67:43361)
0618 22:20:29.727456 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0618 22:20:29.727482 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0618 22:20:29.727505 (+ 23us) server_negotiation.cc:407] Connection header received
0618 22:20:29.727576 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:20:29.727581 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:20:29.727755 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:20:29.727896 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:20:29.728145 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:29.729115 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:29.730528 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:29.731398 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:29.731496 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:29.731635 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0618 22:20:29.732542 (+ 907us) server_negotiation.cc:299] Negotiation successful
0618 22:20:29.732640 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":94,"threads_started":1}
I20260618 22:20:29.733287 14793 tablet_bootstrap.cc:492] T b1193469535d4d2bb5bdaa63c4303273 P a2c7bdc59554485ba027d28fee230475: No bootstrap required, opened a new log
I20260618 22:20:29.733377 14793 ts_tablet_manager.cc:1397] T b1193469535d4d2bb5bdaa63c4303273 P a2c7bdc59554485ba027d28fee230475: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260618 22:20:29.733749 14793 raft_consensus.cc:348] T b1193469535d4d2bb5bdaa63c4303273 P a2c7bdc59554485ba027d28fee230475 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d6036ca6d44f6fb120d8c00a3c3665" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46349 } } peers { permanent_uuid: "222f7b84f00d4e38afcb56c5cc3a1455" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43361 } } peers { permanent_uuid: "a2c7bdc59554485ba027d28fee230475" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34547 } }
I20260618 22:20:29.733865 14793 raft_consensus.cc:374] T b1193469535d4d2bb5bdaa63c4303273 P a2c7bdc59554485ba027d28fee230475 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:20:29.733928 14793 raft_consensus.cc:729] T b1193469535d4d2bb5bdaa63c4303273 P a2c7bdc59554485ba027d28fee230475 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c7bdc59554485ba027d28fee230475, State: Initialized, Role: FOLLOWER
I20260618 22:20:29.734055 14793 consensus_queue.cc:260] T b1193469535d4d2bb5bdaa63c4303273 P a2c7bdc59554485ba027d28fee230475 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d6036ca6d44f6fb120d8c00a3c3665" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46349 } } peers { permanent_uuid: "222f7b84f00d4e38afcb56c5cc3a1455" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43361 } } peers { permanent_uuid: "a2c7bdc59554485ba027d28fee230475" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34547 } }
I20260618 22:20:29.734333 14793 ts_tablet_manager.cc:1428] T b1193469535d4d2bb5bdaa63c4303273 P a2c7bdc59554485ba027d28fee230475: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260618 22:20:29.735428 15105 tablet_service.cc:1467] Processing CreateTablet for tablet b1193469535d4d2bb5bdaa63c4303273 (DEFAULT_TABLE table=test-table [id=9dc6b3402f324d27b92514d7b9957d7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:20:29.735428 14973 tablet_service.cc:1467] Processing CreateTablet for tablet b1193469535d4d2bb5bdaa63c4303273 (DEFAULT_TABLE table=test-table [id=9dc6b3402f324d27b92514d7b9957d7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:20:29.736469 14973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1193469535d4d2bb5bdaa63c4303273. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:20:29.736490 15105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1193469535d4d2bb5bdaa63c4303273. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:20:29.744318 15184 tablet_bootstrap.cc:492] T b1193469535d4d2bb5bdaa63c4303273 P 59d6036ca6d44f6fb120d8c00a3c3665: Bootstrap starting.
I20260618 22:20:29.744318 15185 tablet_bootstrap.cc:492] T b1193469535d4d2bb5bdaa63c4303273 P 222f7b84f00d4e38afcb56c5cc3a1455: Bootstrap starting.
I20260618 22:20:29.747064 15185 tablet_bootstrap.cc:654] T b1193469535d4d2bb5bdaa63c4303273 P 222f7b84f00d4e38afcb56c5cc3a1455: Neither blocks nor log segments found. Creating new log.
I20260618 22:20:29.747121 15184 tablet_bootstrap.cc:654] T b1193469535d4d2bb5bdaa63c4303273 P 59d6036ca6d44f6fb120d8c00a3c3665: Neither blocks nor log segments found. Creating new log.
I20260618 22:20:29.747957 15184 log.cc:824] T b1193469535d4d2bb5bdaa63c4303273 P 59d6036ca6d44f6fb120d8c00a3c3665: Log is configured to *not* fsync() on all Append() calls
I20260618 22:20:29.747957 15185 log.cc:824] T b1193469535d4d2bb5bdaa63c4303273 P 222f7b84f00d4e38afcb56c5cc3a1455: Log is configured to *not* fsync() on all Append() calls
I20260618 22:20:29.749791 15184 tablet_bootstrap.cc:492] T b1193469535d4d2bb5bdaa63c4303273 P 59d6036ca6d44f6fb120d8c00a3c3665: No bootstrap required, opened a new log
I20260618 22:20:29.749845 15185 tablet_bootstrap.cc:492] T b1193469535d4d2bb5bdaa63c4303273 P 222f7b84f00d4e38afcb56c5cc3a1455: No bootstrap required, opened a new log
I20260618 22:20:29.749995 15184 ts_tablet_manager.cc:1397] T b1193469535d4d2bb5bdaa63c4303273 P 59d6036ca6d44f6fb120d8c00a3c3665: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260618 22:20:29.750064 15185 ts_tablet_manager.cc:1397] T b1193469535d4d2bb5bdaa63c4303273 P 222f7b84f00d4e38afcb56c5cc3a1455: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260618 22:20:29.753016 15184 raft_consensus.cc:348] T b1193469535d4d2bb5bdaa63c4303273 P 59d6036ca6d44f6fb120d8c00a3c3665 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d6036ca6d44f6fb120d8c00a3c3665" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46349 } } peers { permanent_uuid: "222f7b84f00d4e38afcb56c5cc3a1455" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43361 } } peers { permanent_uuid: "a2c7bdc59554485ba027d28fee230475" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34547 } }
I20260618 22:20:29.753219 15184 raft_consensus.cc:374] T b1193469535d4d2bb5bdaa63c4303273 P 59d6036ca6d44f6fb120d8c00a3c3665 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:20:29.753101 15185 raft_consensus.cc:348] T b1193469535d4d2bb5bdaa63c4303273 P 222f7b84f00d4e38afcb56c5cc3a1455 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d6036ca6d44f6fb120d8c00a3c3665" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46349 } } peers { permanent_uuid: "222f7b84f00d4e38afcb56c5cc3a1455" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43361 } } peers { permanent_uuid: "a2c7bdc59554485ba027d28fee230475" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34547 } }
I20260618 22:20:29.753283 15184 raft_consensus.cc:729] T b1193469535d4d2bb5bdaa63c4303273 P 59d6036ca6d44f6fb120d8c00a3c3665 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59d6036ca6d44f6fb120d8c00a3c3665, State: Initialized, Role: FOLLOWER
I20260618 22:20:29.753341 15185 raft_consensus.cc:374] T b1193469535d4d2bb5bdaa63c4303273 P 222f7b84f00d4e38afcb56c5cc3a1455 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:20:29.753403 15185 raft_consensus.cc:729] T b1193469535d4d2bb5bdaa63c4303273 P 222f7b84f00d4e38afcb56c5cc3a1455 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 222f7b84f00d4e38afcb56c5cc3a1455, State: Initialized, Role: FOLLOWER
I20260618 22:20:29.753707 15184 consensus_queue.cc:260] T b1193469535d4d2bb5bdaa63c4303273 P 59d6036ca6d44f6fb120d8c00a3c3665 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d6036ca6d44f6fb120d8c00a3c3665" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46349 } } peers { permanent_uuid: "222f7b84f00d4e38afcb56c5cc3a1455" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43361 } } peers { permanent_uuid: "a2c7bdc59554485ba027d28fee230475" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34547 } }
I20260618 22:20:29.753782 15185 consensus_queue.cc:260] T b1193469535d4d2bb5bdaa63c4303273 P 222f7b84f00d4e38afcb56c5cc3a1455 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d6036ca6d44f6fb120d8c00a3c3665" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46349 } } peers { permanent_uuid: "222f7b84f00d4e38afcb56c5cc3a1455" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43361 } } peers { permanent_uuid: "a2c7bdc59554485ba027d28fee230475" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34547 } }
I20260618 22:20:29.754319 15039 heartbeater.cc:503] Master 127.0.98.126:38557 was elected leader, sending a full tablet report...
I20260618 22:20:29.754648 15184 ts_tablet_manager.cc:1428] T b1193469535d4d2bb5bdaa63c4303273 P 59d6036ca6d44f6fb120d8c00a3c3665: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260618 22:20:29.754685 15185 ts_tablet_manager.cc:1428] T b1193469535d4d2bb5bdaa63c4303273 P 222f7b84f00d4e38afcb56c5cc3a1455: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20260618 22:20:29.754966 15171 heartbeater.cc:503] Master 127.0.98.126:38557 was elected leader, sending a full tablet report...
W20260618 22:20:29.759215 15040 tablet.cc:2369] T b1193469535d4d2bb5bdaa63c4303273 P 59d6036ca6d44f6fb120d8c00a3c3665: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:20:29.762811 15188 raft_consensus.cc:482] T b1193469535d4d2bb5bdaa63c4303273 P 59d6036ca6d44f6fb120d8c00a3c3665 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:20:29.762959 15188 raft_consensus.cc:504] T b1193469535d4d2bb5bdaa63c4303273 P 59d6036ca6d44f6fb120d8c00a3c3665 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d6036ca6d44f6fb120d8c00a3c3665" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46349 } } peers { permanent_uuid: "222f7b84f00d4e38afcb56c5cc3a1455" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43361 } } peers { permanent_uuid: "a2c7bdc59554485ba027d28fee230475" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34547 } }
I20260618 22:20:29.763890 15188 leader_election.cc:288] T b1193469535d4d2bb5bdaa63c4303273 P 59d6036ca6d44f6fb120d8c00a3c3665 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 222f7b84f00d4e38afcb56c5cc3a1455 (127.0.98.67:43361), a2c7bdc59554485ba027d28fee230475 (127.0.98.65:34547)
I20260618 22:20:29.767750 15041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:29.764073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:43361 (local address 127.0.98.66:46557)
0618 22:20:29.764180 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:29.764195 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:20:29.764270 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:20:29.764579 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:20:29.764601 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:20:29.764647 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:20:29.764760 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:29.764766 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:29.765586 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:29.765589 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:29.766509 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:29.766518 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:29.767329 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:29.767332 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:29.767441 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:29.767467 (+ 26us) client_negotiation.cc:769] Sending connection context
0618 22:20:29.767549 (+ 82us) client_negotiation.cc:240] Negotiation successful
0618 22:20:29.767618 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"spinlock_wait_cycles":6656}
I20260618 22:20:29.767760 15182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:29.764142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:46557 (local address 127.0.98.67:43361)
0618 22:20:29.764367 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0618 22:20:29.764373 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:20:29.764387 (+ 14us) server_negotiation.cc:407] Connection header received
0618 22:20:29.764428 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:20:29.764431 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:20:29.764476 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:20:29.764555 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:20:29.764890 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:29.765474 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:29.766628 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:29.767214 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:29.767320 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:29.767386 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0618 22:20:29.767573 (+ 187us) server_negotiation.cc:299] Negotiation successful
0618 22:20:29.767638 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
I20260618 22:20:29.768105 14909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:29.764219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:54127 (local address 127.0.98.65:34547)
0618 22:20:29.764374 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0618 22:20:29.764378 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:20:29.764612 (+ 234us) server_negotiation.cc:407] Connection header received
0618 22:20:29.764735 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:20:29.764738 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:20:29.764798 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:20:29.764891 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:20:29.765140 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:29.765725 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:29.766819 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:29.767457 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:29.767572 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:29.767710 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0618 22:20:29.767849 (+ 139us) server_negotiation.cc:299] Negotiation successful
0618 22:20:29.767958 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260618 22:20:29.768344 15125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1193469535d4d2bb5bdaa63c4303273" candidate_uuid: "59d6036ca6d44f6fb120d8c00a3c3665" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "222f7b84f00d4e38afcb56c5cc3a1455" is_pre_election: true
I20260618 22:20:29.768429 15190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:29.764089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:34547 (local address 127.0.98.66:54127)
0618 22:20:29.764524 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:29.764541 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0618 22:20:29.764649 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:20:29.764911 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:20:29.764914 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:20:29.764931 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:20:29.765036 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:29.765042 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:29.765831 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:29.765834 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:29.766690 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:29.766701 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:29.767585 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:29.767589 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:29.767688 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:29.767710 (+ 22us) client_negotiation.cc:769] Sending connection context
0618 22:20:29.768219 (+ 509us) client_negotiation.cc:240] Negotiation successful
0618 22:20:29.768305 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"mutex_wait_us":99,"spinlock_wait_cycles":28288,"thread_start_us":126,"threads_started":1}
I20260618 22:20:29.768615 15125 raft_consensus.cc:2454] T b1193469535d4d2bb5bdaa63c4303273 P 222f7b84f00d4e38afcb56c5cc3a1455 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59d6036ca6d44f6fb120d8c00a3c3665 in term 0.
I20260618 22:20:29.768939 14838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1193469535d4d2bb5bdaa63c4303273" candidate_uuid: "59d6036ca6d44f6fb120d8c00a3c3665" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c7bdc59554485ba027d28fee230475" is_pre_election: true
I20260618 22:20:29.769032 14927 leader_election.cc:302] T b1193469535d4d2bb5bdaa63c4303273 P 59d6036ca6d44f6fb120d8c00a3c3665 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 222f7b84f00d4e38afcb56c5cc3a1455, 59d6036ca6d44f6fb120d8c00a3c3665; no voters:
I20260618 22:20:29.769202 14838 raft_consensus.cc:2454] T b1193469535d4d2bb5bdaa63c4303273 P a2c7bdc59554485ba027d28fee230475 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 59d6036ca6d44f6fb120d8c00a3c3665 in term 0.
I20260618 22:20:29.769311 15188 raft_consensus.cc:2781] T b1193469535d4d2bb5bdaa63c4303273 P 59d6036ca6d44f6fb120d8c00a3c3665 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:20:29.769392 15188 raft_consensus.cc:482] T b1193469535d4d2bb5bdaa63c4303273 P 59d6036ca6d44f6fb120d8c00a3c3665 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:20:29.769460 15188 raft_consensus.cc:3037] T b1193469535d4d2bb5bdaa63c4303273 P 59d6036ca6d44f6fb120d8c00a3c3665 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:20:29.770512 15188 raft_consensus.cc:504] T b1193469535d4d2bb5bdaa63c4303273 P 59d6036ca6d44f6fb120d8c00a3c3665 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d6036ca6d44f6fb120d8c00a3c3665" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46349 } } peers { permanent_uuid: "222f7b84f00d4e38afcb56c5cc3a1455" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43361 } } peers { permanent_uuid: "a2c7bdc59554485ba027d28fee230475" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34547 } }
I20260618 22:20:29.770864 15188 leader_election.cc:288] T b1193469535d4d2bb5bdaa63c4303273 P 59d6036ca6d44f6fb120d8c00a3c3665 [CANDIDATE]: Term 1 election: Requested vote from peers 222f7b84f00d4e38afcb56c5cc3a1455 (127.0.98.67:43361), a2c7bdc59554485ba027d28fee230475 (127.0.98.65:34547)
I20260618 22:20:29.771170 15125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1193469535d4d2bb5bdaa63c4303273" candidate_uuid: "59d6036ca6d44f6fb120d8c00a3c3665" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "222f7b84f00d4e38afcb56c5cc3a1455"
I20260618 22:20:29.771277 14838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1193469535d4d2bb5bdaa63c4303273" candidate_uuid: "59d6036ca6d44f6fb120d8c00a3c3665" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2c7bdc59554485ba027d28fee230475"
I20260618 22:20:29.771277 15125 raft_consensus.cc:3037] T b1193469535d4d2bb5bdaa63c4303273 P 222f7b84f00d4e38afcb56c5cc3a1455 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:20:29.771415 14838 raft_consensus.cc:3037] T b1193469535d4d2bb5bdaa63c4303273 P a2c7bdc59554485ba027d28fee230475 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:20:29.772258 15125 raft_consensus.cc:2454] T b1193469535d4d2bb5bdaa63c4303273 P 222f7b84f00d4e38afcb56c5cc3a1455 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59d6036ca6d44f6fb120d8c00a3c3665 in term 1.
I20260618 22:20:29.772485 14838 raft_consensus.cc:2454] T b1193469535d4d2bb5bdaa63c4303273 P a2c7bdc59554485ba027d28fee230475 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 59d6036ca6d44f6fb120d8c00a3c3665 in term 1.
I20260618 22:20:29.772508 14927 leader_election.cc:302] T b1193469535d4d2bb5bdaa63c4303273 P 59d6036ca6d44f6fb120d8c00a3c3665 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 222f7b84f00d4e38afcb56c5cc3a1455, 59d6036ca6d44f6fb120d8c00a3c3665; no voters:
I20260618 22:20:29.772694 15188 raft_consensus.cc:2781] T b1193469535d4d2bb5bdaa63c4303273 P 59d6036ca6d44f6fb120d8c00a3c3665 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:20:29.772922 15188 raft_consensus.cc:686] T b1193469535d4d2bb5bdaa63c4303273 P 59d6036ca6d44f6fb120d8c00a3c3665 [term 1 LEADER]: Becoming Leader. State: Replica: 59d6036ca6d44f6fb120d8c00a3c3665, State: Running, Role: LEADER
I20260618 22:20:29.773223 15188 consensus_queue.cc:237] T b1193469535d4d2bb5bdaa63c4303273 P 59d6036ca6d44f6fb120d8c00a3c3665 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d6036ca6d44f6fb120d8c00a3c3665" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46349 } } peers { permanent_uuid: "222f7b84f00d4e38afcb56c5cc3a1455" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43361 } } peers { permanent_uuid: "a2c7bdc59554485ba027d28fee230475" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34547 } }
I20260618 22:20:29.776329 14713 catalog_manager.cc:5495] T b1193469535d4d2bb5bdaa63c4303273 P 59d6036ca6d44f6fb120d8c00a3c3665 reported cstate change: term changed from 0 to 1, leader changed from <none> to 59d6036ca6d44f6fb120d8c00a3c3665 (127.0.98.66). New cstate: current_term: 1 leader_uuid: "59d6036ca6d44f6fb120d8c00a3c3665" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d6036ca6d44f6fb120d8c00a3c3665" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46349 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "222f7b84f00d4e38afcb56c5cc3a1455" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2c7bdc59554485ba027d28fee230475" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34547 } health_report { overall_health: UNKNOWN } } }
W20260618 22:20:29.787320 14902 tablet.cc:2369] T b1193469535d4d2bb5bdaa63c4303273 P a2c7bdc59554485ba027d28fee230475: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:20:29.798864 15180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:29.795281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45334 (local address 127.0.98.66:46349)
0618 22:20:29.795417 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0618 22:20:29.795421 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:20:29.795434 (+ 13us) server_negotiation.cc:407] Connection header received
0618 22:20:29.795601 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:20:29.795603 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:20:29.795656 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:20:29.795768 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:20:29.796000 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:29.796597 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:29.797442 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:29.797660 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:29.797772 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:29.798098 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:20:29.798475 (+ 377us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:20:29.798610 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0618 22:20:29.798672 (+ 62us) server_negotiation.cc:299] Negotiation successful
0618 22:20:29.798723 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260618 22:20:29.804386 14838 raft_consensus.cc:1261] T b1193469535d4d2bb5bdaa63c4303273 P a2c7bdc59554485ba027d28fee230475 [term 1 FOLLOWER]: Refusing update from remote peer 59d6036ca6d44f6fb120d8c00a3c3665: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:20:29.804394 15125 raft_consensus.cc:1261] T b1193469535d4d2bb5bdaa63c4303273 P 222f7b84f00d4e38afcb56c5cc3a1455 [term 1 FOLLOWER]: Refusing update from remote peer 59d6036ca6d44f6fb120d8c00a3c3665: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:20:29.805032 15188 consensus_queue.cc:1035] T b1193469535d4d2bb5bdaa63c4303273 P 59d6036ca6d44f6fb120d8c00a3c3665 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2c7bdc59554485ba027d28fee230475" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:20:29.805245 15191 consensus_queue.cc:1035] T b1193469535d4d2bb5bdaa63c4303273 P 59d6036ca6d44f6fb120d8c00a3c3665 [LEADER]: Connected to new peer: Peer: permanent_uuid: "222f7b84f00d4e38afcb56c5cc3a1455" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:20:29.810477 15194 mvcc.cc:205] Tried to move back new op lower bound from 7298339757271810048 to 7298339757153554432. Current Snapshot: MvccSnapshot[applied={T|T < 7298339757271810048}]
I20260618 22:20:29.811124 15195 mvcc.cc:205] Tried to move back new op lower bound from 7298339757271810048 to 7298339757153554432. Current Snapshot: MvccSnapshot[applied={T|T < 7298339757271810048}]
I20260618 22:20:29.818308 14713 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:45162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:20:29.818588 14713 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:45162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:20:29.820731 14713 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dbe151a423ee4337936f2226d65741da: Sending DeleteTablet for 3 replicas of tablet b1193469535d4d2bb5bdaa63c4303273
I20260618 22:20:29.821405 14973 tablet_service.cc:1514] Processing DeleteTablet for tablet b1193469535d4d2bb5bdaa63c4303273 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:20:29 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:45330
I20260618 22:20:29.821399 15105 tablet_service.cc:1514] Processing DeleteTablet for tablet b1193469535d4d2bb5bdaa63c4303273 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:20:29 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:41694
I20260618 22:20:29.821871 15204 tablet_replica.cc:330] T b1193469535d4d2bb5bdaa63c4303273 P 222f7b84f00d4e38afcb56c5cc3a1455: stopping tablet replica
I20260618 22:20:29.822031 14817 tablet_service.cc:1514] Processing DeleteTablet for tablet b1193469535d4d2bb5bdaa63c4303273 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:20:29 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:55734
I20260618 22:20:29.822088 15205 tablet_replica.cc:330] T b1193469535d4d2bb5bdaa63c4303273 P 59d6036ca6d44f6fb120d8c00a3c3665: stopping tablet replica
I20260618 22:20:29.822196 14910 tablet_replica.cc:330] T b1193469535d4d2bb5bdaa63c4303273 P a2c7bdc59554485ba027d28fee230475: stopping tablet replica
I20260618 22:20:29.822296 14910 raft_consensus.cc:2229] T b1193469535d4d2bb5bdaa63c4303273 P a2c7bdc59554485ba027d28fee230475 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:20:29.822348 15205 raft_consensus.cc:2229] T b1193469535d4d2bb5bdaa63c4303273 P 59d6036ca6d44f6fb120d8c00a3c3665 [term 1 LEADER]: Raft consensus shutting down.
I20260618 22:20:29.822434 14910 raft_consensus.cc:2258] T b1193469535d4d2bb5bdaa63c4303273 P a2c7bdc59554485ba027d28fee230475 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:20:29.822346 15204 raft_consensus.cc:2229] T b1193469535d4d2bb5bdaa63c4303273 P 222f7b84f00d4e38afcb56c5cc3a1455 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:20:29.822664 15205 raft_consensus.cc:2258] T b1193469535d4d2bb5bdaa63c4303273 P 59d6036ca6d44f6fb120d8c00a3c3665 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:20:29.822763 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 14751
I20260618 22:20:29.822788 15204 raft_consensus.cc:2258] T b1193469535d4d2bb5bdaa63c4303273 P 222f7b84f00d4e38afcb56c5cc3a1455 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:20:29.824465 15205 ts_tablet_manager.cc:1905] T b1193469535d4d2bb5bdaa63c4303273 P 59d6036ca6d44f6fb120d8c00a3c3665: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:20:29.828399 15205 ts_tablet_manager.cc:1918] T b1193469535d4d2bb5bdaa63c4303273 P 59d6036ca6d44f6fb120d8c00a3c3665: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:20:29.828501 15205 log.cc:1196] T b1193469535d4d2bb5bdaa63c4303273 P 59d6036ca6d44f6fb120d8c00a3c3665: Deleting WAL directory at /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-1/wal/wals/b1193469535d4d2bb5bdaa63c4303273
I20260618 22:20:29.828855 15205 ts_tablet_manager.cc:1939] T b1193469535d4d2bb5bdaa63c4303273 P 59d6036ca6d44f6fb120d8c00a3c3665: Deleting consensus metadata
I20260618 22:20:29.828986 15204 ts_tablet_manager.cc:1905] T b1193469535d4d2bb5bdaa63c4303273 P 222f7b84f00d4e38afcb56c5cc3a1455: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:20:29.830013 14697 catalog_manager.cc:4841] TS 59d6036ca6d44f6fb120d8c00a3c3665 (127.0.98.66:46349): tablet b1193469535d4d2bb5bdaa63c4303273 (table test-table [id=9dc6b3402f324d27b92514d7b9957d7d]) successfully deleted
W20260618 22:20:29.831128 14699 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.65:34547 (error 108)
I20260618 22:20:29.831691 15204 ts_tablet_manager.cc:1918] T b1193469535d4d2bb5bdaa63c4303273 P 222f7b84f00d4e38afcb56c5cc3a1455: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:20:29.831857 15204 log.cc:1196] T b1193469535d4d2bb5bdaa63c4303273 P 222f7b84f00d4e38afcb56c5cc3a1455: Deleting WAL directory at /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-2/wal/wals/b1193469535d4d2bb5bdaa63c4303273
I20260618 22:20:29.831935 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 14905
I20260618 22:20:29.832252 15181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:29.831932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:34547 (local address 127.0.0.1:55740)
0618 22:20:29.832082 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:29.832161 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.65:34547: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260618 22:20:29.832890 15204 ts_tablet_manager.cc:1939] T b1193469535d4d2bb5bdaa63c4303273 P 222f7b84f00d4e38afcb56c5cc3a1455: Deleting consensus metadata
W20260618 22:20:29.833293 14699 catalog_manager.cc:4584] TS a2c7bdc59554485ba027d28fee230475 (127.0.98.65:34547): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b1193469535d4d2bb5bdaa63c4303273: Network error: Client connection negotiation failed: client connection to 127.0.98.65:34547: connect: Connection refused (error 111)
I20260618 22:20:29.834029 14698 catalog_manager.cc:4841] TS 222f7b84f00d4e38afcb56c5cc3a1455 (127.0.98.67:43361): tablet b1193469535d4d2bb5bdaa63c4303273 (table test-table [id=9dc6b3402f324d27b92514d7b9957d7d]) successfully deleted
I20260618 22:20:29.837821 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 15042
I20260618 22:20:29.843045 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 14680
I20260618 22:20:29.849050 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:38557
--webserver_interface=127.0.98.126
--webserver_port=44585
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:43207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:38557
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:20:29.955171 15207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:20:29.955431 15207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:20:29.955511 15207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:20:29.959425 15207 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:20:29.959501 15207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:20:29.959539 15207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:20:29.959597 15207 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:20:29.959637 15207 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:20:29.964632 15207 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:43207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:38557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:38557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=44585
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15207
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:20:29.965545 15207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:20:29.966473 15207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:20:29.974386 15213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:20:29.974399 15215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:20:29.974421 15212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:20:29.974668 15207 server_base.cc:1034] running on GCE node
I20260618 22:20:29.975207 15207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:20:29.976186 15207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:20:29.977339 15207 hybrid_clock.cc:648] HybridClock initialized: now 1781821229977317 us; error 35 us; skew 500 ppm
Jun 18 22:20:29 dist-test-slave-g6r4 krb5kdc[14133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821229, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:20:29.979851 15207 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260618 22:20:29.980660 15207 webserver.cc:460] Webserver started at http://127.0.98.126:44585/ using document root <none> and password file <none>
I20260618 22:20:29.981171 15207 fs_manager.cc:359] Metadata directory not provided
I20260618 22:20:29.981258 15207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:20:29.984687 15207 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:20:29.986656 15222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:20:29.987709 15207 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:20:29.987838 15207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/master-0/wal
uuid: "dbe151a423ee4337936f2226d65741da"
format_stamp: "Formatted at 2026-06-18 22:20:27 on dist-test-slave-g6r4"
server_key: "31a3e970244938294a9a6f0a346c5376"
server_key_iv: "ae14f4c650bcd0940dbaee4523a3337b"
server_key_version: "encryptionkey@0"
I20260618 22:20:29.988329 15207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:20:30.008592 15207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:20:30.011262 15207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:20:30.011466 15207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:20:30.018266 15207 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:38557
I20260618 22:20:30.018569 15274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:38557 every 8 connection(s)
I20260618 22:20:30.019284 15207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
I20260618 22:20:30.023351 15275 sys_catalog.cc:263] Verifying existing consensus state
I20260618 22:20:30.024824 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 15207
I20260618 22:20:30.024857 15275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbe151a423ee4337936f2226d65741da: Bootstrap starting.
I20260618 22:20:30.025323 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:34547
--local_ip_for_outbound_sockets=127.0.98.65
--tserver_master_addrs=127.0.98.126:38557
--webserver_port=45317
--webserver_interface=127.0.98.65
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:43207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 22:20:30.042173 15275 log.cc:824] T 00000000000000000000000000000000 P dbe151a423ee4337936f2226d65741da: Log is configured to *not* fsync() on all Append() calls
I20260618 22:20:30.054235 15275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbe151a423ee4337936f2226d65741da: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260618 22:20:30.054636 15275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbe151a423ee4337936f2226d65741da: Bootstrap complete.
I20260618 22:20:30.058979 15275 raft_consensus.cc:348] T 00000000000000000000000000000000 P dbe151a423ee4337936f2226d65741da [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbe151a423ee4337936f2226d65741da" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38557 } }
I20260618 22:20:30.059392 15275 raft_consensus.cc:729] T 00000000000000000000000000000000 P dbe151a423ee4337936f2226d65741da [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbe151a423ee4337936f2226d65741da, State: Initialized, Role: FOLLOWER
I20260618 22:20:30.059954 15275 consensus_queue.cc:260] T 00000000000000000000000000000000 P dbe151a423ee4337936f2226d65741da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbe151a423ee4337936f2226d65741da" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38557 } }
I20260618 22:20:30.060108 15275 raft_consensus.cc:388] T 00000000000000000000000000000000 P dbe151a423ee4337936f2226d65741da [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:20:30.060202 15275 raft_consensus.cc:482] T 00000000000000000000000000000000 P dbe151a423ee4337936f2226d65741da [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:20:30.060326 15275 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dbe151a423ee4337936f2226d65741da [term 2 FOLLOWER]: Advancing to term 3
I20260618 22:20:30.062297 15275 raft_consensus.cc:504] T 00000000000000000000000000000000 P dbe151a423ee4337936f2226d65741da [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbe151a423ee4337936f2226d65741da" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38557 } }
I20260618 22:20:30.062630 15275 leader_election.cc:302] T 00000000000000000000000000000000 P dbe151a423ee4337936f2226d65741da [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dbe151a423ee4337936f2226d65741da; no voters:
I20260618 22:20:30.062971 15275 leader_election.cc:288] T 00000000000000000000000000000000 P dbe151a423ee4337936f2226d65741da [CANDIDATE]: Term 3 election: Requested vote from peers
I20260618 22:20:30.063079 15279 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dbe151a423ee4337936f2226d65741da [term 3 FOLLOWER]: Leader election won for term 3
I20260618 22:20:30.063298 15279 raft_consensus.cc:686] T 00000000000000000000000000000000 P dbe151a423ee4337936f2226d65741da [term 3 LEADER]: Becoming Leader. State: Replica: dbe151a423ee4337936f2226d65741da, State: Running, Role: LEADER
I20260618 22:20:30.063591 15279 consensus_queue.cc:237] T 00000000000000000000000000000000 P dbe151a423ee4337936f2226d65741da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbe151a423ee4337936f2226d65741da" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38557 } }
I20260618 22:20:30.064201 15275 sys_catalog.cc:564] T 00000000000000000000000000000000 P dbe151a423ee4337936f2226d65741da [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:20:30.064559 15281 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbe151a423ee4337936f2226d65741da [sys.catalog]: SysCatalogTable state changed. Reason: New leader dbe151a423ee4337936f2226d65741da. Latest consensus state: current_term: 3 leader_uuid: "dbe151a423ee4337936f2226d65741da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbe151a423ee4337936f2226d65741da" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38557 } } }
I20260618 22:20:30.064690 15281 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbe151a423ee4337936f2226d65741da [sys.catalog]: This master's current role is: LEADER
I20260618 22:20:30.064879 15280 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbe151a423ee4337936f2226d65741da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "dbe151a423ee4337936f2226d65741da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbe151a423ee4337936f2226d65741da" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 38557 } } }
I20260618 22:20:30.064975 15280 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbe151a423ee4337936f2226d65741da [sys.catalog]: This master's current role is: LEADER
I20260618 22:20:30.065053 15283 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:20:30.070514 15283 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:20:30.071241 15283 catalog_manager.cc:1259] Loaded cluster ID: 42893856c82e4dcbb1a581319025cfb1
I20260618 22:20:30.071338 15283 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260618 22:20:30.072161 15283 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f5bc99ed9ad google::LogMessage::Fail() at ??:0
@ 0x7f5bc99f088a google::LogMessage::SendToLog() at ??:0
@ 0x7f5bc99ed4de google::LogMessage::Flush() at ??:0
@ 0x7f5bc99eea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f5bd432cfdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f5bd432dd2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f5bd432a579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f5bd436e640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f5bcbc3f3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f5bcbc3fc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f5bcbc413f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f5bcbc30512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f5bcd2746db start_thread at ??:0
@ 0x7f5bc84eb61f clone at ??:0
W20260618 22:20:30.171610 15277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:20:30.171880 15277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:20:30.171964 15277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:20:30.176337 15277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:20:30.176393 15277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:20:30.176453 15277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260618 22:20:30.181622 15277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:43207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:34547
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=45317
--tserver_master_addrs=127.0.98.126:38557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15277
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:20:30.182662 15277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:20:30.183638 15277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:20:30.193054 15316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:20:30.193063 15317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:20:30.193068 15319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:20:30.193456 15277 server_base.cc:1034] running on GCE node
I20260618 22:20:30.193794 15277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:20:30.194430 15277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:20:30.195607 15277 hybrid_clock.cc:648] HybridClock initialized: now 1781821230195579 us; error 48 us; skew 500 ppm
Jun 18 22:20:30 dist-test-slave-g6r4 krb5kdc[14133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821230, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:20:30.198407 15277 init.cc:377] Logged in from keytab as kudu/127.0.98.65@KRBTEST.COM (short username kudu)
I20260618 22:20:30.199199 15277 webserver.cc:460] Webserver started at http://127.0.98.65:45317/ using document root <none> and password file <none>
I20260618 22:20:30.199786 15277 fs_manager.cc:359] Metadata directory not provided
I20260618 22:20:30.199877 15277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:20:30.203823 15277 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:20:30.206012 15326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:20:30.207216 15277 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:20:30.207347 15277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-0/wal
uuid: "a2c7bdc59554485ba027d28fee230475"
format_stamp: "Formatted at 2026-06-18 22:20:27 on dist-test-slave-g6r4"
server_key: "9a934605ba3da4a73d2275f4cb40e86e"
server_key_iv: "3d1359fb93e157e37f591124f86586f5"
server_key_version: "encryptionkey@0"
I20260618 22:20:30.207885 15277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:20:30.223346 15277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:20:30.226336 15277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:20:30.226536 15277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:20:30.227082 15277 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:20:30.228111 15333 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260618 22:20:30.233006 15277 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260618 22:20:30.233098 15277 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260618 22:20:30.233180 15277 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260618 22:20:30.234839 15277 ts_tablet_manager.cc:610] Registered 1 tablets
I20260618 22:20:30.234910 15277 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260618 22:20:30.235188 15333 tablet_bootstrap.cc:492] T b1193469535d4d2bb5bdaa63c4303273 P a2c7bdc59554485ba027d28fee230475: Bootstrap starting.
I20260618 22:20:30.245550 15277 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:34547
I20260618 22:20:30.246142 15440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:34547 every 8 connection(s)
I20260618 22:20:30.246738 15277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
I20260618 22:20:30.252344 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 15277
I20260618 22:20:30.252751 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:46349
--local_ip_for_outbound_sockets=127.0.98.66
--tserver_master_addrs=127.0.98.126:38557
--webserver_port=33247
--webserver_interface=127.0.98.66
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:43207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 22:20:30.256371 15333 log.cc:824] T b1193469535d4d2bb5bdaa63c4303273 P a2c7bdc59554485ba027d28fee230475: Log is configured to *not* fsync() on all Append() calls
I20260618 22:20:30.264884 15333 tablet_bootstrap.cc:492] T b1193469535d4d2bb5bdaa63c4303273 P a2c7bdc59554485ba027d28fee230475: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260618 22:20:30.265298 15333 tablet_bootstrap.cc:492] T b1193469535d4d2bb5bdaa63c4303273 P a2c7bdc59554485ba027d28fee230475: Bootstrap complete.
I20260618 22:20:30.266716 15333 ts_tablet_manager.cc:1397] T b1193469535d4d2bb5bdaa63c4303273 P a2c7bdc59554485ba027d28fee230475: Time spent bootstrapping tablet: real 0.032s user 0.023s sys 0.008s
I20260618 22:20:30.269587 15333 raft_consensus.cc:348] T b1193469535d4d2bb5bdaa63c4303273 P a2c7bdc59554485ba027d28fee230475 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d6036ca6d44f6fb120d8c00a3c3665" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46349 } } peers { permanent_uuid: "222f7b84f00d4e38afcb56c5cc3a1455" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43361 } } peers { permanent_uuid: "a2c7bdc59554485ba027d28fee230475" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34547 } }
I20260618 22:20:30.270145 15333 raft_consensus.cc:729] T b1193469535d4d2bb5bdaa63c4303273 P a2c7bdc59554485ba027d28fee230475 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2c7bdc59554485ba027d28fee230475, State: Initialized, Role: FOLLOWER
I20260618 22:20:30.270637 15333 consensus_queue.cc:260] T b1193469535d4d2bb5bdaa63c4303273 P a2c7bdc59554485ba027d28fee230475 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d6036ca6d44f6fb120d8c00a3c3665" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46349 } } peers { permanent_uuid: "222f7b84f00d4e38afcb56c5cc3a1455" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43361 } } peers { permanent_uuid: "a2c7bdc59554485ba027d28fee230475" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34547 } }
I20260618 22:20:30.271734 15333 ts_tablet_manager.cc:1428] T b1193469535d4d2bb5bdaa63c4303273 P a2c7bdc59554485ba027d28fee230475: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260618 22:20:30.292500 15443 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:20:30.248629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38557 (local address 127.0.98.65:40817)
0618 22:20:30.249107 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:30.249160 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0618 22:20:30.249998 (+ 838us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:20:30.291588 (+ 41590us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38557: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":153,"threads_started":1}
W20260618 22:20:30.293085 15441 heartbeater.cc:644] Failed to heartbeat to 127.0.98.126:38557 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.126:38557: Client connection negotiation failed: client connection to 127.0.98.126:38557: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260618 22:20:30.293778 15443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:30.293521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38557 (local address 127.0.98.65:36353)
0618 22:20:30.293650 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:30.293719 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260618 22:20:30.294483 15443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:30.294243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38557 (local address 127.0.98.65:41179)
0618 22:20:30.294350 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:30.294424 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260618 22:20:30.294665 15441 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260618 22:20:30.363965 15444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:20:30.364197 15444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:20:30.364238 15444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:20:30.368134 15444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:20:30.368182 15444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:20:30.368235 15444 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260618 22:20:30.373291 15444 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:43207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:46349
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=33247
--tserver_master_addrs=127.0.98.126:38557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:20:30.374243 15444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:20:30.375135 15444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:20:30.384289 15454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:20:30.384357 15452 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:20:30.384373 15451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:20:30.384438 15444 server_base.cc:1034] running on GCE node
I20260618 22:20:30.384946 15444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:20:30.385569 15444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:20:30.386730 15444 hybrid_clock.cc:648] HybridClock initialized: now 1781821230386708 us; error 24 us; skew 500 ppm
Jun 18 22:20:30 dist-test-slave-g6r4 krb5kdc[14133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821230, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:20:30.389310 15444 init.cc:377] Logged in from keytab as kudu/127.0.98.66@KRBTEST.COM (short username kudu)
I20260618 22:20:30.390239 15444 webserver.cc:460] Webserver started at http://127.0.98.66:33247/ using document root <none> and password file <none>
I20260618 22:20:30.390758 15444 fs_manager.cc:359] Metadata directory not provided
I20260618 22:20:30.390815 15444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:20:30.394460 15444 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:20:30.396423 15461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:20:30.397570 15444 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:20:30.397684 15444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-1/wal
uuid: "59d6036ca6d44f6fb120d8c00a3c3665"
format_stamp: "Formatted at 2026-06-18 22:20:28 on dist-test-slave-g6r4"
server_key: "7cf93c9fd250cd36e81ba505b89d34a6"
server_key_iv: "7564773e47f86984d2b4d5a87d2729ed"
server_key_version: "encryptionkey@0"
I20260618 22:20:30.398226 15444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:20:30.412139 15444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:20:30.414793 15444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:20:30.414994 15444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:20:30.415582 15444 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:20:30.416343 15444 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:20:30.416394 15444 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:20:30.416461 15444 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:20:30.416510 15444 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:20:30.425414 15444 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:46349
I20260618 22:20:30.425518 15574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:46349 every 8 connection(s)
I20260618 22:20:30.426694 15444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
I20260618 22:20:30.430506 15577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:30.428842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38557 (local address 127.0.98.66:43365)
0618 22:20:30.429655 (+ 813us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:30.429765 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":511,"threads_started":1}
I20260618 22:20:30.430689 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 15444
W20260618 22:20:30.430941 15575 heartbeater.cc:644] Failed to heartbeat to 127.0.98.126:38557 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.126:38557: Client connection negotiation failed: client connection to 127.0.98.126:38557: connect: Connection refused (error 111)
I20260618 22:20:30.431149 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:43361
--local_ip_for_outbound_sockets=127.0.98.67
--tserver_master_addrs=127.0.98.126:38557
--webserver_port=37883
--webserver_interface=127.0.98.67
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:43207
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 22:20:30.431571 15577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:30.431330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38557 (local address 127.0.98.66:39991)
0618 22:20:30.431457 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:30.431521 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260618 22:20:30.432156 15577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:30.431981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38557 (local address 127.0.98.66:49963)
0618 22:20:30.432073 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:30.432111 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260618 22:20:30.432327 15575 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260618 22:20:30.534623 15578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:20:30.534875 15578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:20:30.534953 15578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:20:30.538794 15578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:20:30.538870 15578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:20:30.538960 15578 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260618 22:20:30.543793 15578 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:43207
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:43361
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=37883
--tserver_master_addrs=127.0.98.126:38557
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:20:30.544661 15578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:20:30.545547 15578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:20:30.554317 15586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:20:30.554468 15578 server_base.cc:1034] running on GCE node
W20260618 22:20:30.554353 15584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:20:30.554354 15583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:20:30.554925 15578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:20:30.555493 15578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:20:30.556648 15578 hybrid_clock.cc:648] HybridClock initialized: now 1781821230556638 us; error 23 us; skew 500 ppm
Jun 18 22:20:30 dist-test-slave-g6r4 krb5kdc[14133](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821230, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:20:30.559156 15578 init.cc:377] Logged in from keytab as kudu/127.0.98.67@KRBTEST.COM (short username kudu)
I20260618 22:20:30.559962 15578 webserver.cc:460] Webserver started at http://127.0.98.67:37883/ using document root <none> and password file <none>
I20260618 22:20:30.560462 15578 fs_manager.cc:359] Metadata directory not provided
I20260618 22:20:30.560549 15578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:20:30.563993 15578 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:20:30.566007 15593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:20:30.567158 15578 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:20:30.567289 15578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-2/wal
uuid: "222f7b84f00d4e38afcb56c5cc3a1455"
format_stamp: "Formatted at 2026-06-18 22:20:28 on dist-test-slave-g6r4"
server_key: "8b92e37e1fdb89f9145107e6724a1e83"
server_key_iv: "d1ba7ee082abf672580920f1705806b0"
server_key_version: "encryptionkey@0"
I20260618 22:20:30.567773 15578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:20:30.592166 15578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:20:30.595023 15578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:20:30.595209 15578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:20:30.595762 15578 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:20:30.596534 15578 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:20:30.596603 15578 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:20:30.596674 15578 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:20:30.596724 15578 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:20:30.605296 15578 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:43361
I20260618 22:20:30.605319 15706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:43361 every 8 connection(s)
I20260618 22:20:30.606293 15578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
I20260618 22:20:30.607272 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 15578
I20260618 22:20:30.610206 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:30.608480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38557 (local address 127.0.98.67:43347)
0618 22:20:30.608898 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:30.609009 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":111,"threads_started":1}
W20260618 22:20:30.610723 15707 heartbeater.cc:644] Failed to heartbeat to 127.0.98.126:38557 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.126:38557: Client connection negotiation failed: client connection to 127.0.98.126:38557: connect: Connection refused (error 111)
I20260618 22:20:30.611387 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:30.611113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38557 (local address 127.0.98.67:57187)
0618 22:20:30.611239 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:30.611335 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260618 22:20:30.612007 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:30.611795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38557 (local address 127.0.98.67:58855)
0618 22:20:30.611896 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:30.611941 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260618 22:20:30.612160 15707 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260618 22:20:31.295585 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:31.295202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38557 (local address 127.0.98.65:41937)
0618 22:20:31.295423 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:31.295481 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":70,"threads_started":1}
I20260618 22:20:31.412969 15712 raft_consensus.cc:482] T b1193469535d4d2bb5bdaa63c4303273 P a2c7bdc59554485ba027d28fee230475 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:20:31.413216 15712 raft_consensus.cc:504] T b1193469535d4d2bb5bdaa63c4303273 P a2c7bdc59554485ba027d28fee230475 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d6036ca6d44f6fb120d8c00a3c3665" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46349 } } peers { permanent_uuid: "222f7b84f00d4e38afcb56c5cc3a1455" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43361 } } peers { permanent_uuid: "a2c7bdc59554485ba027d28fee230475" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34547 } }
I20260618 22:20:31.414486 15712 leader_election.cc:288] T b1193469535d4d2bb5bdaa63c4303273 P a2c7bdc59554485ba027d28fee230475 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59d6036ca6d44f6fb120d8c00a3c3665 (127.0.98.66:46349), 222f7b84f00d4e38afcb56c5cc3a1455 (127.0.98.67:43361)
Jun 18 22:20:31 dist-test-slave-g6r4 krb5kdc[14133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821230, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for kudu/127.0.98.66@KRBTEST.COM
Jun 18 22:20:31 dist-test-slave-g6r4 krb5kdc[14133](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821230, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for kudu/127.0.98.67@KRBTEST.COM
I20260618 22:20:31.428292 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:31.414702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:43361 (local address 127.0.98.65:58643)
0618 22:20:31.415110 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:31.415124 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:20:31.415221 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:20:31.416634 (+ 1413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:20:31.416640 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:20:31.417158 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:20:31.417625 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:31.417639 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:31.418960 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:31.418965 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:31.419885 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:31.419894 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:31.420586 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:31.420589 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:31.420797 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:31.421540 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:20:31.421558 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:20:31.423766 (+ 2208us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:20:31.426014 (+ 2248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:20:31.426017 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:20:31.426020 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:20:31.426260 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:20:31.426648 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:20:31.426651 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:20:31.426653 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:20:31.426745 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:20:31.427479 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:20:31.427482 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:20:31.427577 (+ 95us) client_negotiation.cc:769] Sending connection context
0618 22:20:31.427736 (+ 159us) client_negotiation.cc:240] Negotiation successful
0618 22:20:31.428078 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"mutex_wait_us":211,"spinlock_wait_cycles":3456,"thread_start_us":72,"threads_started":1}
I20260618 22:20:31.428375 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:31.414935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:40329 (local address 127.0.98.66:46349)
0618 22:20:31.415330 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0618 22:20:31.415340 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0618 22:20:31.415370 (+ 30us) server_negotiation.cc:407] Connection header received
0618 22:20:31.415964 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:20:31.415970 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:20:31.416273 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:20:31.416583 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:20:31.417797 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:31.418805 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:31.420014 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:31.420470 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:31.420646 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:31.423603 (+ 2957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:20:31.423609 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:20:31.423624 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:20:31.423660 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:20:31.425622 (+ 1962us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:31.426280 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:31.426287 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:31.426292 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:31.426425 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:31.426808 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:31.426812 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:31.426814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:31.427200 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:20:31.427338 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0618 22:20:31.427865 (+ 527us) server_negotiation.cc:299] Negotiation successful
0618 22:20:31.428168 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":53,"threads_started":1}
I20260618 22:20:31.428721 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:31.414702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:46349 (local address 127.0.98.65:40329)
0618 22:20:31.414840 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:31.414857 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0618 22:20:31.414966 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:20:31.416635 (+ 1669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:20:31.416640 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:20:31.417158 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:20:31.417625 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:31.417639 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:31.418953 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:31.418966 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:31.419882 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:31.419891 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:31.420594 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:31.420598 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:31.420800 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:31.421553 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:20:31.421569 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:20:31.423354 (+ 1785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:20:31.425740 (+ 2386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:20:31.425746 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:20:31.425758 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:20:31.426163 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:20:31.426537 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:20:31.426540 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:20:31.426542 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:20:31.426711 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:20:31.427325 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:20:31.427331 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:20:31.427549 (+ 218us) client_negotiation.cc:769] Sending connection context
0618 22:20:31.427736 (+ 187us) client_negotiation.cc:240] Negotiation successful
0618 22:20:31.428168 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":312,"spinlock_wait_cycles":15232}
I20260618 22:20:31.429216 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:31.414899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:58643 (local address 127.0.98.67:43361)
0618 22:20:31.415226 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0618 22:20:31.415238 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0618 22:20:31.415276 (+ 38us) server_negotiation.cc:407] Connection header received
0618 22:20:31.415917 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:20:31.415923 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:20:31.416273 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:20:31.416583 (+ 310us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:20:31.417802 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:31.418806 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:31.420011 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:31.420461 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:31.420643 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:31.423944 (+ 3301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:20:31.423948 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:20:31.423959 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:20:31.423993 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:20:31.425872 (+ 1879us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:31.426374 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:31.426379 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:31.426384 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:31.426542 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:31.426857 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:31.426861 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:31.426862 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:31.427305 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:20:31.427422 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0618 22:20:31.428725 (+ 1303us) server_negotiation.cc:299] Negotiation successful
0618 22:20:31.429008 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":124,"threads_started":1}
W20260618 22:20:31.432092 15328 leader_election.cc:341] T b1193469535d4d2bb5bdaa63c4303273 P a2c7bdc59554485ba027d28fee230475 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 222f7b84f00d4e38afcb56c5cc3a1455 (127.0.98.67:43361): Not found: Tablet not found: b1193469535d4d2bb5bdaa63c4303273
W20260618 22:20:31.432261 15327 leader_election.cc:341] T b1193469535d4d2bb5bdaa63c4303273 P a2c7bdc59554485ba027d28fee230475 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 59d6036ca6d44f6fb120d8c00a3c3665 (127.0.98.66:46349): Not found: Tablet not found: b1193469535d4d2bb5bdaa63c4303273
I20260618 22:20:31.432341 15327 leader_election.cc:302] T b1193469535d4d2bb5bdaa63c4303273 P a2c7bdc59554485ba027d28fee230475 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2c7bdc59554485ba027d28fee230475; no voters: 222f7b84f00d4e38afcb56c5cc3a1455, 59d6036ca6d44f6fb120d8c00a3c3665
I20260618 22:20:31.432583 15712 raft_consensus.cc:2726] T b1193469535d4d2bb5bdaa63c4303273 P a2c7bdc59554485ba027d28fee230475 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260618 22:20:31.433359 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:31.432984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38557 (local address 127.0.98.66:35871)
0618 22:20:31.433210 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:31.433264 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":72,"threads_started":1}
I20260618 22:20:31.612978 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:31.612619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38557 (local address 127.0.98.67:52089)
0618 22:20:31.612841 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:31.612892 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":77,"threads_started":1}
I20260618 22:20:32.296674 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:32.296281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38557 (local address 127.0.98.65:35933)
0618 22:20:32.296530 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:32.296583 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":73,"threads_started":1}
I20260618 22:20:32.434646 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:32.434078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38557 (local address 127.0.98.66:49929)
0618 22:20:32.434400 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:32.434502 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":82,"threads_started":1}
I20260618 22:20:32.614041 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:32.613623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38557 (local address 127.0.98.67:54927)
0618 22:20:32.613839 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:32.613890 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":66,"threads_started":1}
I20260618 22:20:33.116447 15721 raft_consensus.cc:482] T b1193469535d4d2bb5bdaa63c4303273 P a2c7bdc59554485ba027d28fee230475 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:20:33.116568 15721 raft_consensus.cc:504] T b1193469535d4d2bb5bdaa63c4303273 P a2c7bdc59554485ba027d28fee230475 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d6036ca6d44f6fb120d8c00a3c3665" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46349 } } peers { permanent_uuid: "222f7b84f00d4e38afcb56c5cc3a1455" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43361 } } peers { permanent_uuid: "a2c7bdc59554485ba027d28fee230475" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34547 } }
I20260618 22:20:33.117048 15721 leader_election.cc:288] T b1193469535d4d2bb5bdaa63c4303273 P a2c7bdc59554485ba027d28fee230475 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59d6036ca6d44f6fb120d8c00a3c3665 (127.0.98.66:46349), 222f7b84f00d4e38afcb56c5cc3a1455 (127.0.98.67:43361)
W20260618 22:20:33.117818 15328 leader_election.cc:341] T b1193469535d4d2bb5bdaa63c4303273 P a2c7bdc59554485ba027d28fee230475 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 222f7b84f00d4e38afcb56c5cc3a1455 (127.0.98.67:43361): Not found: Tablet not found: b1193469535d4d2bb5bdaa63c4303273
W20260618 22:20:33.117954 15327 leader_election.cc:341] T b1193469535d4d2bb5bdaa63c4303273 P a2c7bdc59554485ba027d28fee230475 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 59d6036ca6d44f6fb120d8c00a3c3665 (127.0.98.66:46349): Not found: Tablet not found: b1193469535d4d2bb5bdaa63c4303273
I20260618 22:20:33.118045 15327 leader_election.cc:302] T b1193469535d4d2bb5bdaa63c4303273 P a2c7bdc59554485ba027d28fee230475 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2c7bdc59554485ba027d28fee230475; no voters: 222f7b84f00d4e38afcb56c5cc3a1455, 59d6036ca6d44f6fb120d8c00a3c3665
I20260618 22:20:33.118211 15721 raft_consensus.cc:2726] T b1193469535d4d2bb5bdaa63c4303273 P a2c7bdc59554485ba027d28fee230475 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260618 22:20:33.297768 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:33.297393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38557 (local address 127.0.98.65:51937)
0618 22:20:33.297625 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:33.297677 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":74,"threads_started":1}
I20260618 22:20:33.435894 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:33.435395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38557 (local address 127.0.98.66:50627)
0618 22:20:33.435682 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:33.435768 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":104,"threads_started":1}
I20260618 22:20:33.615106 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:33.614697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38557 (local address 127.0.98.67:57103)
0618 22:20:33.614972 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:33.615022 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":126,"threads_started":1}
I20260618 22:20:34.298909 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:34.298481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38557 (local address 127.0.98.65:55995)
0618 22:20:34.298764 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:34.298812 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":103,"threads_started":1}
I20260618 22:20:34.437057 15726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:34.436597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38557 (local address 127.0.98.66:46035)
0618 22:20:34.436894 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:34.436967 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":97,"threads_started":1}
I20260618 22:20:34.616125 15727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:34.615763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38557 (local address 127.0.98.67:56173)
0618 22:20:34.615988 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:34.616039 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":69,"threads_started":1}
I20260618 22:20:35.300258 15728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:35.299824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38557 (local address 127.0.98.65:37183)
0618 22:20:35.300104 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:35.300154 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":68,"threads_started":1}
I20260618 22:20:35.438172 15729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:35.437753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38557 (local address 127.0.98.66:39255)
0618 22:20:35.438010 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:35.438061 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":91,"threads_started":1}
I20260618 22:20:35.617367 15730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:35.616896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38557 (local address 127.0.98.67:46163)
0618 22:20:35.617205 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:35.617266 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":92,"threads_started":1}
I20260618 22:20:35.964349 15731 raft_consensus.cc:482] T b1193469535d4d2bb5bdaa63c4303273 P a2c7bdc59554485ba027d28fee230475 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:20:35.964514 15731 raft_consensus.cc:504] T b1193469535d4d2bb5bdaa63c4303273 P a2c7bdc59554485ba027d28fee230475 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d6036ca6d44f6fb120d8c00a3c3665" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46349 } } peers { permanent_uuid: "222f7b84f00d4e38afcb56c5cc3a1455" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43361 } } peers { permanent_uuid: "a2c7bdc59554485ba027d28fee230475" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34547 } }
I20260618 22:20:35.964922 15731 leader_election.cc:288] T b1193469535d4d2bb5bdaa63c4303273 P a2c7bdc59554485ba027d28fee230475 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59d6036ca6d44f6fb120d8c00a3c3665 (127.0.98.66:46349), 222f7b84f00d4e38afcb56c5cc3a1455 (127.0.98.67:43361)
W20260618 22:20:35.965677 15328 leader_election.cc:341] T b1193469535d4d2bb5bdaa63c4303273 P a2c7bdc59554485ba027d28fee230475 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 222f7b84f00d4e38afcb56c5cc3a1455 (127.0.98.67:43361): Not found: Tablet not found: b1193469535d4d2bb5bdaa63c4303273
W20260618 22:20:35.965761 15327 leader_election.cc:341] T b1193469535d4d2bb5bdaa63c4303273 P a2c7bdc59554485ba027d28fee230475 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 59d6036ca6d44f6fb120d8c00a3c3665 (127.0.98.66:46349): Not found: Tablet not found: b1193469535d4d2bb5bdaa63c4303273
I20260618 22:20:35.965803 15328 leader_election.cc:302] T b1193469535d4d2bb5bdaa63c4303273 P a2c7bdc59554485ba027d28fee230475 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2c7bdc59554485ba027d28fee230475; no voters: 222f7b84f00d4e38afcb56c5cc3a1455, 59d6036ca6d44f6fb120d8c00a3c3665
I20260618 22:20:35.965962 15731 raft_consensus.cc:2726] T b1193469535d4d2bb5bdaa63c4303273 P a2c7bdc59554485ba027d28fee230475 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260618 22:20:36.301405 15732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:36.300987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38557 (local address 127.0.98.65:53127)
0618 22:20:36.301252 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:36.301308 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":90,"threads_started":1}
I20260618 22:20:36.439291 15733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:36.438904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38557 (local address 127.0.98.66:55601)
0618 22:20:36.439152 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:36.439205 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":72,"threads_started":1}
I20260618 22:20:36.618525 15734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:36.618144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38557 (local address 127.0.98.67:58551)
0618 22:20:36.618383 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:36.618438 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":69,"threads_started":1}
I20260618 22:20:37.302515 15735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:37.302127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38557 (local address 127.0.98.65:49811)
0618 22:20:37.302367 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:37.302422 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":75,"threads_started":1}
I20260618 22:20:37.440537 15736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:37.440131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38557 (local address 127.0.98.66:37443)
0618 22:20:37.440401 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:37.440455 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":64,"threads_started":1}
I20260618 22:20:37.619580 15737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:37.619177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38557 (local address 127.0.98.67:54307)
0618 22:20:37.619444 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:37.619495 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":69,"threads_started":1}
I20260618 22:20:38.303686 15738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:38.303199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38557 (local address 127.0.98.65:36443)
0618 22:20:38.303464 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:38.303523 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":75,"threads_started":1}
I20260618 22:20:38.441743 15739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:38.441350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38557 (local address 127.0.98.66:41567)
0618 22:20:38.441600 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:38.441655 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":93,"threads_started":1}
I20260618 22:20:38.620707 15740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:38.620319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38557 (local address 127.0.98.67:34883)
0618 22:20:38.620561 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:38.620618 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":70,"threads_started":1}
I20260618 22:20:39.304867 15741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:39.304468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38557 (local address 127.0.98.65:43257)
0618 22:20:39.304720 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:39.304776 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":79,"threads_started":1}
I20260618 22:20:39.442879 15742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:39.442492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38557 (local address 127.0.98.66:35445)
0618 22:20:39.442734 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:39.442793 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":73,"threads_started":1}
I20260618 22:20:39.621963 15743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:39.621444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38557 (local address 127.0.98.67:42187)
0618 22:20:39.621741 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:39.621807 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":87,"threads_started":1}
I20260618 22:20:40.306316 15744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:40.305871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38557 (local address 127.0.98.65:51603)
0618 22:20:40.306165 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:40.306220 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":72,"threads_started":1}
I20260618 22:20:40.444016 15745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:40.443650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38557 (local address 127.0.98.66:55423)
0618 22:20:40.443879 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:40.443931 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":69,"threads_started":1}
I20260618 22:20:40.593288 15746 raft_consensus.cc:482] T b1193469535d4d2bb5bdaa63c4303273 P a2c7bdc59554485ba027d28fee230475 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:20:40.593416 15746 raft_consensus.cc:504] T b1193469535d4d2bb5bdaa63c4303273 P a2c7bdc59554485ba027d28fee230475 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "59d6036ca6d44f6fb120d8c00a3c3665" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 46349 } } peers { permanent_uuid: "222f7b84f00d4e38afcb56c5cc3a1455" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 43361 } } peers { permanent_uuid: "a2c7bdc59554485ba027d28fee230475" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34547 } }
I20260618 22:20:40.593782 15746 leader_election.cc:288] T b1193469535d4d2bb5bdaa63c4303273 P a2c7bdc59554485ba027d28fee230475 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 59d6036ca6d44f6fb120d8c00a3c3665 (127.0.98.66:46349), 222f7b84f00d4e38afcb56c5cc3a1455 (127.0.98.67:43361)
W20260618 22:20:40.594601 15328 leader_election.cc:341] T b1193469535d4d2bb5bdaa63c4303273 P a2c7bdc59554485ba027d28fee230475 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 222f7b84f00d4e38afcb56c5cc3a1455 (127.0.98.67:43361): Not found: Tablet not found: b1193469535d4d2bb5bdaa63c4303273
W20260618 22:20:40.594708 15327 leader_election.cc:341] T b1193469535d4d2bb5bdaa63c4303273 P a2c7bdc59554485ba027d28fee230475 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 59d6036ca6d44f6fb120d8c00a3c3665 (127.0.98.66:46349): Not found: Tablet not found: b1193469535d4d2bb5bdaa63c4303273
I20260618 22:20:40.594782 15328 leader_election.cc:302] T b1193469535d4d2bb5bdaa63c4303273 P a2c7bdc59554485ba027d28fee230475 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a2c7bdc59554485ba027d28fee230475; no voters: 222f7b84f00d4e38afcb56c5cc3a1455, 59d6036ca6d44f6fb120d8c00a3c3665
I20260618 22:20:40.594933 15746 raft_consensus.cc:2726] T b1193469535d4d2bb5bdaa63c4303273 P a2c7bdc59554485ba027d28fee230475 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260618 22:20:40.623149 15747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:40.622775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38557 (local address 127.0.98.67:33063)
0618 22:20:40.623004 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:40.623058 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":68,"threads_started":1}
I20260618 22:20:41.307338 15748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:41.306973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38557 (local address 127.0.98.65:57181)
0618 22:20:41.307197 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:41.307248 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":73,"threads_started":1}
I20260618 22:20:41.445190 15749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:41.444775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38557 (local address 127.0.98.66:55823)
0618 22:20:41.445038 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:41.445095 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":95,"threads_started":1}
I20260618 22:20:41.624339 15750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:41.623934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38557 (local address 127.0.98.67:44489)
0618 22:20:41.624193 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:41.624248 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":89,"threads_started":1}
I20260618 22:20:42.308339 15751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:42.307964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38557 (local address 127.0.98.65:43559)
0618 22:20:42.308200 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:42.308250 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":84,"threads_started":1}
I20260618 22:20:42.446334 15752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:42.445954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38557 (local address 127.0.98.66:49683)
0618 22:20:42.446194 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:42.446245 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":79,"threads_started":1}
I20260618 22:20:42.625365 15753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:42.624972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38557 (local address 127.0.98.67:44105)
0618 22:20:42.625225 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:42.625274 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":86,"threads_started":1}
I20260618 22:20:43.309381 15754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:43.308998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38557 (local address 127.0.98.65:47921)
0618 22:20:43.309235 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:43.309288 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":72,"threads_started":1}
I20260618 22:20:43.447453 15755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:43.447091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38557 (local address 127.0.98.66:41711)
0618 22:20:43.447313 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:43.447362 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":76,"threads_started":1}
I20260618 22:20:43.626434 15756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:43.626060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38557 (local address 127.0.98.67:42733)
0618 22:20:43.626290 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:43.626343 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":67,"threads_started":1}
I20260618 22:20:44.310600 15757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:44.310201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38557 (local address 127.0.98.65:60269)
0618 22:20:44.310454 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:44.310509 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":89,"threads_started":1}
I20260618 22:20:44.448545 15758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:44.448150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38557 (local address 127.0.98.66:47471)
0618 22:20:44.448402 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:44.448456 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":77,"threads_started":1}
I20260618 22:20:44.627490 15759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:44.627108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38557 (local address 127.0.98.67:40795)
0618 22:20:44.627350 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:44.627401 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":92,"threads_started":1}
I20260618 22:20:45.312000 15760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:45.311590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38557 (local address 127.0.98.65:46903)
0618 22:20:45.311825 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:45.311899 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":86,"threads_started":1}
I20260618 22:20:45.449734 15761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:45.449374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:38557 (local address 127.0.98.66:56347)
0618 22:20:45.449590 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:45.449644 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:38557: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":68,"threads_started":1}
I20260618 22:20:45.609730 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 15277
I20260618 22:20:45.615242 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 15444
I20260618 22:20:45.620296 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 15578
2026-06-18T22:20:45Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19396 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:20:45 dist-test-slave-g6r4 krb5kdc[15764](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:20:45 dist-test-slave-g6r4 krb5kdc[15764](info): set up 2 sockets
Jun 18 22:20:45 dist-test-slave-g6r4 krb5kdc[15764](info): commencing operation
krb5kdc: starting...
W20260618 22:20:46.674034 393 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/krb5kdc/test-user.keytab.
Jun 18 22:20:46 dist-test-slave-g6r4 krb5kdc[15764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:20:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:20:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260618 22:20:46.790979 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:39001
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:38325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:39001
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:20:46.899598 15777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:20:46.899964 15777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:20:46.900054 15777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:20:46.906955 15777 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260618 22:20:46.907028 15777 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260618 22:20:46.907054 15777 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260618 22:20:46.907087 15777 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:20:46.907126 15777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:20:46.907164 15777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:20:46.907199 15777 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:20:46.907234 15777 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:20:46.912513 15777 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:38325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:39001
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:39001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15777
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:20:46.913491 15777 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:20:46.914526 15777 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:20:46.922744 15785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:20:46.922749 15782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:20:46.922864 15777 server_base.cc:1034] running on GCE node
W20260618 22:20:46.922771 15783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:20:46.923674 15777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:20:46.924767 15777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:20:46.925973 15777 hybrid_clock.cc:648] HybridClock initialized: now 1781821246925962 us; error 30 us; skew 500 ppm
Jun 18 22:20:46 dist-test-slave-g6r4 krb5kdc[15764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821246, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:20:46.928529 15777 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260618 22:20:46.929391 15777 webserver.cc:460] Webserver started at http://127.0.98.126:37291/ using document root <none> and password file <none>
I20260618 22:20:46.929991 15777 fs_manager.cc:359] Metadata directory not provided
I20260618 22:20:46.930080 15777 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:20:46.930297 15777 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:20:46.932210 15777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/master-0/data/instance:
uuid: "56d1566a923f47dca11d653707ba497a"
format_stamp: "Formatted at 2026-06-18 22:20:46 on dist-test-slave-g6r4"
server_key: "339cc53340c9db76e65087b522b03ba8"
server_key_iv: "9a85caacb642db088086e741d0d79546"
server_key_version: "encryptionkey@0"
I20260618 22:20:46.932766 15777 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/master-0/wal/instance:
uuid: "56d1566a923f47dca11d653707ba497a"
format_stamp: "Formatted at 2026-06-18 22:20:46 on dist-test-slave-g6r4"
server_key: "339cc53340c9db76e65087b522b03ba8"
server_key_iv: "9a85caacb642db088086e741d0d79546"
server_key_version: "encryptionkey@0"
I20260618 22:20:46.936252 15777 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260618 22:20:46.938611 15792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:20:46.939766 15777 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260618 22:20:46.939903 15777 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/master-0/wal
uuid: "56d1566a923f47dca11d653707ba497a"
format_stamp: "Formatted at 2026-06-18 22:20:46 on dist-test-slave-g6r4"
server_key: "339cc53340c9db76e65087b522b03ba8"
server_key_iv: "9a85caacb642db088086e741d0d79546"
server_key_version: "encryptionkey@0"
I20260618 22:20:46.940011 15777 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:20:46.968879 15777 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:20:46.971725 15777 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:20:46.971923 15777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:20:46.978845 15777 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:39001
I20260618 22:20:46.979143 15844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:39001 every 8 connection(s)
I20260618 22:20:46.979795 15777 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
I20260618 22:20:46.982398 15845 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:20:46.987287 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 15777
I20260618 22:20:46.987424 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/master-0/wal/instance
I20260618 22:20:46.987447 15845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56d1566a923f47dca11d653707ba497a: Bootstrap starting.
I20260618 22:20:46.987718 393 external_mini_cluster.cc:1428] Setting key 19b6ef196ae3f15ccc7aad9f089a1182
I20260618 22:20:46.989773 15845 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56d1566a923f47dca11d653707ba497a: Neither blocks nor log segments found. Creating new log.
I20260618 22:20:46.990501 15845 log.cc:824] T 00000000000000000000000000000000 P 56d1566a923f47dca11d653707ba497a: Log is configured to *not* fsync() on all Append() calls
I20260618 22:20:46.992383 15845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56d1566a923f47dca11d653707ba497a: No bootstrap required, opened a new log
Jun 18 22:20:46 dist-test-slave-g6r4 krb5kdc[15764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:20:46.996730 15845 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56d1566a923f47dca11d653707ba497a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d1566a923f47dca11d653707ba497a" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39001 } }
I20260618 22:20:46.996948 15845 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56d1566a923f47dca11d653707ba497a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:20:46.997057 15845 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56d1566a923f47dca11d653707ba497a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56d1566a923f47dca11d653707ba497a, State: Initialized, Role: FOLLOWER
I20260618 22:20:46.997464 15845 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56d1566a923f47dca11d653707ba497a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d1566a923f47dca11d653707ba497a" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39001 } }
I20260618 22:20:46.997611 15845 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56d1566a923f47dca11d653707ba497a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:20:46.997690 15845 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56d1566a923f47dca11d653707ba497a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:20:46.997797 15845 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56d1566a923f47dca11d653707ba497a [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:20:46.998603 15845 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56d1566a923f47dca11d653707ba497a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d1566a923f47dca11d653707ba497a" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39001 } }
I20260618 22:20:46.998910 15845 leader_election.cc:302] T 00000000000000000000000000000000 P 56d1566a923f47dca11d653707ba497a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56d1566a923f47dca11d653707ba497a; no voters:
I20260618 22:20:46.999203 15845 leader_election.cc:288] T 00000000000000000000000000000000 P 56d1566a923f47dca11d653707ba497a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:20:46.999264 15850 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56d1566a923f47dca11d653707ba497a [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:20:46.999464 15850 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56d1566a923f47dca11d653707ba497a [term 1 LEADER]: Becoming Leader. State: Replica: 56d1566a923f47dca11d653707ba497a, State: Running, Role: LEADER
I20260618 22:20:46.999783 15850 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56d1566a923f47dca11d653707ba497a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d1566a923f47dca11d653707ba497a" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39001 } }
I20260618 22:20:47.000401 15845 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56d1566a923f47dca11d653707ba497a [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:20:47.001308 15851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56d1566a923f47dca11d653707ba497a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56d1566a923f47dca11d653707ba497a. Latest consensus state: current_term: 1 leader_uuid: "56d1566a923f47dca11d653707ba497a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d1566a923f47dca11d653707ba497a" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39001 } } }
I20260618 22:20:47.001611 15852 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56d1566a923f47dca11d653707ba497a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56d1566a923f47dca11d653707ba497a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d1566a923f47dca11d653707ba497a" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39001 } } }
I20260618 22:20:47.001720 15851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56d1566a923f47dca11d653707ba497a [sys.catalog]: This master's current role is: LEADER
I20260618 22:20:47.001793 15852 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56d1566a923f47dca11d653707ba497a [sys.catalog]: This master's current role is: LEADER
I20260618 22:20:47.002643 15848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:46.988994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43362 (local address 127.0.98.126:39001)
0618 22:20:46.989378 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0618 22:20:46.989393 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0618 22:20:46.989426 (+ 33us) server_negotiation.cc:407] Connection header received
0618 22:20:46.990117 (+ 691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:20:46.990136 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:20:46.990443 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:20:46.990767 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:20:46.991549 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:46.992527 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:46.993672 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:46.994192 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:46.994315 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:46.997034 (+ 2719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:20:46.997039 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:20:46.997051 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:20:46.997076 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:20:46.999159 (+ 2083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:46.999669 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:46.999674 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:46.999679 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:46.999821 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:47.000114 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:47.000119 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:47.000121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:47.000521 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:20:47.000675 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0618 22:20:47.001128 (+ 453us) server_negotiation.cc:299] Negotiation successful
0618 22:20:47.001456 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":248,"thread_start_us":136,"threads_started":1}
W20260618 22:20:47.004977 15865 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 56d1566a923f47dca11d653707ba497a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 22:20:47.005088 15865 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 22:20:47.005182 15860 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:20:47.005834 15860 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:20:47.011137 15860 catalog_manager.cc:1347] Generated new cluster ID: 1146874a6ad643a9a163dfb9a1560f69
I20260618 22:20:47.011196 15860 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:20:47.022389 15860 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:20:47.023268 15860 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:20:47.030346 15860 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 56d1566a923f47dca11d653707ba497a: Generated new TSK 0
I20260618 22:20:47.030961 15860 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab.
I20260618 22:20:47.065439 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:39001
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:38325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:20:47.170822 15871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:20:47.171088 15871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:20:47.171185 15871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:20:47.175066 15871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:20:47.175150 15871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:20:47.175240 15871 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260618 22:20:47.180225 15871 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:38325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:39001
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15871
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:20:47.181174 15871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:20:47.182142 15871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:20:47.191051 15876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:20:47.191044 15877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:20:47.191082 15871 server_base.cc:1034] running on GCE node
W20260618 22:20:47.191047 15879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:20:47.191636 15871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:20:47.192248 15871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:20:47.193430 15871 hybrid_clock.cc:648] HybridClock initialized: now 1781821247193413 us; error 28 us; skew 500 ppm
Jun 18 22:20:47 dist-test-slave-g6r4 krb5kdc[15764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821247, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:20:47.196022 15871 init.cc:377] Logged in from keytab as kudu/127.0.98.65@KRBTEST.COM (short username kudu)
I20260618 22:20:47.196801 15871 webserver.cc:460] Webserver started at http://127.0.98.65:35901/ using document root <none> and password file <none>
I20260618 22:20:47.197319 15871 fs_manager.cc:359] Metadata directory not provided
I20260618 22:20:47.197407 15871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:20:47.197619 15871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:20:47.199510 15871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-0/data/instance:
uuid: "ae5c1a50e67f4aed8830a55970810e8e"
format_stamp: "Formatted at 2026-06-18 22:20:47 on dist-test-slave-g6r4"
server_key: "0eea59f8f5fe730b371cff829746fe6f"
server_key_iv: "681f3e6a2126ea0e2d2a237f13dc42d0"
server_key_version: "encryptionkey@0"
I20260618 22:20:47.200031 15871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-0/wal/instance:
uuid: "ae5c1a50e67f4aed8830a55970810e8e"
format_stamp: "Formatted at 2026-06-18 22:20:47 on dist-test-slave-g6r4"
server_key: "0eea59f8f5fe730b371cff829746fe6f"
server_key_iv: "681f3e6a2126ea0e2d2a237f13dc42d0"
server_key_version: "encryptionkey@0"
I20260618 22:20:47.203640 15871 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260618 22:20:47.205827 15886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:20:47.206946 15871 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:20:47.207086 15871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-0/wal
uuid: "ae5c1a50e67f4aed8830a55970810e8e"
format_stamp: "Formatted at 2026-06-18 22:20:47 on dist-test-slave-g6r4"
server_key: "0eea59f8f5fe730b371cff829746fe6f"
server_key_iv: "681f3e6a2126ea0e2d2a237f13dc42d0"
server_key_version: "encryptionkey@0"
I20260618 22:20:47.207190 15871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:20:47.227291 15871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:20:47.230124 15871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:20:47.230325 15871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:20:47.230924 15871 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:20:47.231752 15871 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:20:47.231820 15871 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:20:47.231884 15871 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:20:47.231938 15871 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:20:47.240005 15871 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:33093
I20260618 22:20:47.240068 15999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:33093 every 8 connection(s)
I20260618 22:20:47.240844 15871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
I20260618 22:20:47.241480 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 15871
I20260618 22:20:47.241762 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-0/wal/instance
I20260618 22:20:47.242131 393 external_mini_cluster.cc:1428] Setting key 24c073d2dfd459211d36d5a8bd6cd445
Jun 18 22:20:47 dist-test-slave-g6r4 krb5kdc[15764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821247, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.66@KRBTEST.COM" created.
I20260618 22:20:47.257362 15848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:47.242780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:33945 (local address 127.0.98.126:39001)
0618 22:20:47.242946 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0618 22:20:47.242951 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:20:47.243884 (+ 933us) server_negotiation.cc:407] Connection header received
0618 22:20:47.245531 (+ 1647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:20:47.245535 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:20:47.245582 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:20:47.245676 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:20:47.247399 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:47.247994 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:47.250046 (+ 2052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:47.250274 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:47.250356 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:47.253245 (+ 2889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:20:47.253249 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:20:47.253251 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:20:47.253274 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:20:47.254926 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:47.255532 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:47.255539 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:47.255544 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:47.255656 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:47.256108 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:47.256111 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:47.256112 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:47.256286 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:20:47.256361 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0618 22:20:47.257004 (+ 643us) server_negotiation.cc:299] Negotiation successful
0618 22:20:47.257147 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260618 22:20:47.258153 16002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:47.243200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39001 (local address 127.0.98.65:33945)
0618 22:20:47.243708 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:47.243763 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0618 22:20:47.245281 (+ 1518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:20:47.245932 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:20:47.245946 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:20:47.246513 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:20:47.247233 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:47.247256 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:47.248137 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:47.248141 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:47.249890 (+ 1749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:47.249932 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:47.250438 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:47.250443 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:47.250665 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:47.251345 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:20:47.251373 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:20:47.253093 (+ 1720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:20:47.255030 (+ 1937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:20:47.255034 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:20:47.255044 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:20:47.255432 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:20:47.255807 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:20:47.255811 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:20:47.255813 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:20:47.256004 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:20:47.256386 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:20:47.256395 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:20:47.256710 (+ 315us) client_negotiation.cc:769] Sending connection context
0618 22:20:47.256927 (+ 217us) client_negotiation.cc:240] Negotiation successful
0618 22:20:47.257194 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":175,"threads_started":1}
I20260618 22:20:47.259334 16000 heartbeater.cc:347] Connected to a master server at 127.0.98.126:39001
I20260618 22:20:47.259562 16000 heartbeater.cc:464] Registering TS with master...
I20260618 22:20:47.259995 16000 heartbeater.cc:511] Master 127.0.98.126:39001 requested a full tablet report, sending...
I20260618 22:20:47.261420 15809 ts_manager.cc:194] Registered new tserver with Master: ae5c1a50e67f4aed8830a55970810e8e (127.0.98.65:33093)
I20260618 22:20:47.262549 15809 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.65@KRBTEST.COM'} at 127.0.98.65:33945
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab.
I20260618 22:20:47.272464 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:39001
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:38325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:20:47.377802 16005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:20:47.378103 16005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:20:47.378181 16005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:20:47.382135 16005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:20:47.382203 16005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:20:47.382293 16005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260618 22:20:47.387341 16005 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:38325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:39001
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16005
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:20:47.388320 16005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:20:47.389211 16005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:20:47.399029 16013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:20:47.399049 16010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:20:47.399168 16005 server_base.cc:1034] running on GCE node
W20260618 22:20:47.399049 16011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:20:47.399681 16005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:20:47.400377 16005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:20:47.401548 16005 hybrid_clock.cc:648] HybridClock initialized: now 1781821247401527 us; error 33 us; skew 500 ppm
Jun 18 22:20:47 dist-test-slave-g6r4 krb5kdc[15764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821247, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:20:47.404310 16005 init.cc:377] Logged in from keytab as kudu/127.0.98.66@KRBTEST.COM (short username kudu)
I20260618 22:20:47.405225 16005 webserver.cc:460] Webserver started at http://127.0.98.66:39183/ using document root <none> and password file <none>
I20260618 22:20:47.405779 16005 fs_manager.cc:359] Metadata directory not provided
I20260618 22:20:47.405867 16005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:20:47.406102 16005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:20:47.408118 16005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-1/data/instance:
uuid: "610a350e9c644e61bd41f8646ed9f7c7"
format_stamp: "Formatted at 2026-06-18 22:20:47 on dist-test-slave-g6r4"
server_key: "be8fc677a8c3224a0f362f5a111672d8"
server_key_iv: "9375ad4ed12a38ad90323ab04f36043b"
server_key_version: "encryptionkey@0"
I20260618 22:20:47.408672 16005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-1/wal/instance:
uuid: "610a350e9c644e61bd41f8646ed9f7c7"
format_stamp: "Formatted at 2026-06-18 22:20:47 on dist-test-slave-g6r4"
server_key: "be8fc677a8c3224a0f362f5a111672d8"
server_key_iv: "9375ad4ed12a38ad90323ab04f36043b"
server_key_version: "encryptionkey@0"
I20260618 22:20:47.412606 16005 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.006s
I20260618 22:20:47.415150 16020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:20:47.416368 16005 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260618 22:20:47.416512 16005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-1/wal
uuid: "610a350e9c644e61bd41f8646ed9f7c7"
format_stamp: "Formatted at 2026-06-18 22:20:47 on dist-test-slave-g6r4"
server_key: "be8fc677a8c3224a0f362f5a111672d8"
server_key_iv: "9375ad4ed12a38ad90323ab04f36043b"
server_key_version: "encryptionkey@0"
I20260618 22:20:47.416628 16005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:20:47.433830 16005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:20:47.436651 16005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:20:47.436851 16005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:20:47.437423 16005 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:20:47.438299 16005 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:20:47.438367 16005 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:20:47.438434 16005 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:20:47.438482 16005 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:20:47.447721 16005 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:39395
I20260618 22:20:47.447754 16133 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:39395 every 8 connection(s)
I20260618 22:20:47.448603 16005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
Jun 18 22:20:47 dist-test-slave-g6r4 krb5kdc[15764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821247, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:20:47.458161 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 16005
I20260618 22:20:47.458307 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-1/wal/instance
I20260618 22:20:47.458679 393 external_mini_cluster.cc:1428] Setting key 94a5ec5d82e90860251c05703b3c58f2
I20260618 22:20:47.461781 15848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:47.450383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:48547 (local address 127.0.98.126:39001)
0618 22:20:47.450545 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0618 22:20:47.450550 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:20:47.451369 (+ 819us) server_negotiation.cc:407] Connection header received
0618 22:20:47.452538 (+ 1169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:20:47.452541 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:20:47.452591 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:20:47.452665 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:20:47.453768 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:47.454349 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:47.455433 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:47.455663 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:47.455737 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:47.458059 (+ 2322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:20:47.458064 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:20:47.458067 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:20:47.458094 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:20:47.459526 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:47.460157 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:47.460161 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:47.460162 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:47.460250 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:47.460565 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:47.460569 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:47.460571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:47.460839 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:20:47.460954 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0618 22:20:47.461386 (+ 432us) server_negotiation.cc:299] Negotiation successful
0618 22:20:47.461527 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260618 22:20:47.462667 16136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:47.450782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39001 (local address 127.0.98.66:48547)
0618 22:20:47.451219 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:47.451262 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0618 22:20:47.452352 (+ 1090us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:20:47.452817 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:20:47.452823 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:20:47.453170 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:20:47.453642 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:47.453656 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:47.454449 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:47.454451 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:47.455330 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:47.455339 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:47.455780 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:47.455782 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:47.455919 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:47.456421 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:20:47.456436 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:20:47.457848 (+ 1412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:20:47.459650 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:20:47.459656 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:20:47.459669 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:20:47.460031 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:20:47.460342 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:20:47.460344 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:20:47.460345 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:20:47.460474 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:20:47.460941 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:20:47.460946 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:20:47.461163 (+ 217us) client_negotiation.cc:769] Sending connection context
0618 22:20:47.461315 (+ 152us) client_negotiation.cc:240] Negotiation successful
0618 22:20:47.461553 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":104,"threads_started":1}
I20260618 22:20:47.463768 16134 heartbeater.cc:347] Connected to a master server at 127.0.98.126:39001
I20260618 22:20:47.464006 16134 heartbeater.cc:464] Registering TS with master...
I20260618 22:20:47.464468 16134 heartbeater.cc:511] Master 127.0.98.126:39001 requested a full tablet report, sending...
I20260618 22:20:47.465469 15809 ts_manager.cc:194] Registered new tserver with Master: 610a350e9c644e61bd41f8646ed9f7c7 (127.0.98.66:39395)
I20260618 22:20:47.466070 15809 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.66@KRBTEST.COM'} at 127.0.98.66:48547
WARNING: no policy specified for kudu/127.0.98.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab.
I20260618 22:20:47.484891 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:39001
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:38325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:20:47.591872 16139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:20:47.592118 16139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:20:47.592159 16139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:20:47.596189 16139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:20:47.596247 16139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:20:47.596330 16139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260618 22:20:47.601560 16139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:38325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:39001
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:20:47.602762 16139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:20:47.603729 16139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:20:47.613256 16147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:20:47.613358 16144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:20:47.613261 16145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:20:47.613312 16139 server_base.cc:1034] running on GCE node
I20260618 22:20:47.613952 16139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:20:47.614638 16139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:20:47.615839 16139 hybrid_clock.cc:648] HybridClock initialized: now 1781821247615818 us; error 29 us; skew 500 ppm
Jun 18 22:20:47 dist-test-slave-g6r4 krb5kdc[15764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821247, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:20:47.618597 16139 init.cc:377] Logged in from keytab as kudu/127.0.98.67@KRBTEST.COM (short username kudu)
I20260618 22:20:47.619436 16139 webserver.cc:460] Webserver started at http://127.0.98.67:39675/ using document root <none> and password file <none>
I20260618 22:20:47.620059 16139 fs_manager.cc:359] Metadata directory not provided
I20260618 22:20:47.620147 16139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:20:47.620383 16139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:20:47.622416 16139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-2/data/instance:
uuid: "139063e1765148b181db5d13fdf5ff54"
format_stamp: "Formatted at 2026-06-18 22:20:47 on dist-test-slave-g6r4"
server_key: "0482da59d59c1f4c97da2193113e0cc9"
server_key_iv: "2fe7273eae39923eda6ccc37090ead2b"
server_key_version: "encryptionkey@0"
I20260618 22:20:47.622974 16139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-2/wal/instance:
uuid: "139063e1765148b181db5d13fdf5ff54"
format_stamp: "Formatted at 2026-06-18 22:20:47 on dist-test-slave-g6r4"
server_key: "0482da59d59c1f4c97da2193113e0cc9"
server_key_iv: "2fe7273eae39923eda6ccc37090ead2b"
server_key_version: "encryptionkey@0"
I20260618 22:20:47.626796 16139 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260618 22:20:47.629063 16154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:20:47.630280 16139 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260618 22:20:47.630406 16139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-2/wal
uuid: "139063e1765148b181db5d13fdf5ff54"
format_stamp: "Formatted at 2026-06-18 22:20:47 on dist-test-slave-g6r4"
server_key: "0482da59d59c1f4c97da2193113e0cc9"
server_key_iv: "2fe7273eae39923eda6ccc37090ead2b"
server_key_version: "encryptionkey@0"
I20260618 22:20:47.630510 16139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:20:47.641960 16139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:20:47.645208 16139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:20:47.645394 16139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:20:47.645996 16139 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:20:47.646756 16139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:20:47.646801 16139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:20:47.646885 16139 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:20:47.646920 16139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:20:47.654855 16139 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:44555
I20260618 22:20:47.654927 16267 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:44555 every 8 connection(s)
I20260618 22:20:47.655714 16139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
I20260618 22:20:47.661055 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 16139
I20260618 22:20:47.661181 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-2/wal/instance
I20260618 22:20:47.661437 393 external_mini_cluster.cc:1428] Setting key 2ea8f073ffb63566bdf00bb93b1426e3
Jun 18 22:20:47 dist-test-slave-g6r4 krb5kdc[15764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821247, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:20:47.674850 15848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:47.663499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:39585 (local address 127.0.98.126:39001)
0618 22:20:47.663652 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0618 22:20:47.663657 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:20:47.663673 (+ 16us) server_negotiation.cc:407] Connection header received
0618 22:20:47.663720 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:20:47.663725 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:20:47.663785 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:20:47.663882 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:20:47.665637 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:47.666223 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:47.668105 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:47.668318 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:47.668396 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:47.671202 (+ 2806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:20:47.671205 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:20:47.671207 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:20:47.671228 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:20:47.672606 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:47.673119 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:47.673124 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:47.673126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:47.673211 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:47.673537 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:47.673541 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:47.673543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:47.673719 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:20:47.673819 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0618 22:20:47.674465 (+ 646us) server_negotiation.cc:299] Negotiation successful
0618 22:20:47.674608 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260618 22:20:47.675686 16270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:47.658334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39001 (local address 127.0.98.67:39585)
0618 22:20:47.658865 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:47.658931 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0618 22:20:47.660738 (+ 1807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:20:47.664250 (+ 3512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:20:47.664263 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:20:47.664811 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:20:47.665484 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:47.665502 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:47.666348 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:47.666355 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:47.667961 (+ 1606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:47.667976 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:47.668461 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:47.668467 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:47.668705 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:47.669339 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:20:47.669366 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:20:47.671019 (+ 1653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:20:47.672736 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:20:47.672741 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:20:47.672755 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:20:47.673002 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:20:47.673301 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:20:47.673305 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:20:47.673307 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:20:47.673439 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:20:47.673829 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:20:47.673837 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:20:47.674171 (+ 334us) client_negotiation.cc:769] Sending connection context
0618 22:20:47.674389 (+ 218us) client_negotiation.cc:240] Negotiation successful
0618 22:20:47.674711 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":99,"threads_started":1}
I20260618 22:20:47.676687 16268 heartbeater.cc:347] Connected to a master server at 127.0.98.126:39001
I20260618 22:20:47.676931 16268 heartbeater.cc:464] Registering TS with master...
I20260618 22:20:47.677381 16268 heartbeater.cc:511] Master 127.0.98.126:39001 requested a full tablet report, sending...
I20260618 22:20:47.678349 15809 ts_manager.cc:194] Registered new tserver with Master: 139063e1765148b181db5d13fdf5ff54 (127.0.98.67:44555)
I20260618 22:20:47.678879 15809 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.67@KRBTEST.COM'} at 127.0.98.67:39585
I20260618 22:20:47.687062 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 22:20:47.697214 15848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:47.688985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43370 (local address 127.0.98.126:39001)
0618 22:20:47.689132 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0618 22:20:47.689139 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:20:47.689253 (+ 114us) server_negotiation.cc:407] Connection header received
0618 22:20:47.689354 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:20:47.689359 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:20:47.689413 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:20:47.689513 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:20:47.690156 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:47.691105 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:47.691879 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:47.692322 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:47.692442 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:47.693694 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:20:47.693699 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:20:47.693704 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:20:47.693755 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:20:47.695466 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:47.695862 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:47.695867 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:47.695872 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:47.695965 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:20:47.696210 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:20:47.696214 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:20:47.696218 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:20:47.696524 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:20:47.696673 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0618 22:20:47.696846 (+ 173us) server_negotiation.cc:299] Negotiation successful
0618 22:20:47.696976 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260618 22:20:47.700248 15809 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43370:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 22:20:47.705642 15809 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:20:47.721767 16281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:47.715431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:39395 (local address 127.0.0.1:42926)
0618 22:20:47.715733 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:47.715748 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:20:47.715836 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:20:47.716975 (+ 1139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:20:47.716981 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:20:47.717004 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:20:47.717192 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:47.717202 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:47.718697 (+ 1495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:47.718702 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:47.720074 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:47.720092 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:47.721243 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:47.721247 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:47.721359 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:47.721409 (+ 50us) client_negotiation.cc:769] Sending connection context
0618 22:20:47.721493 (+ 84us) client_negotiation.cc:240] Negotiation successful
0618 22:20:47.721619 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":41,"spinlock_wait_cycles":4480,"thread_start_us":92,"threads_started":1}
I20260618 22:20:47.723237 16282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:47.715632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42926 (local address 127.0.98.66:39395)
0618 22:20:47.716529 (+ 897us) server_negotiation.cc:206] Beginning negotiation
0618 22:20:47.716541 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0618 22:20:47.716559 (+ 18us) server_negotiation.cc:407] Connection header received
0618 22:20:47.716631 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:20:47.716635 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:20:47.716800 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:20:47.717212 (+ 412us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:20:47.717332 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:47.718520 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:47.720207 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:47.721042 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:47.722059 (+ 1017us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:47.722937 (+ 878us) server_negotiation.cc:1035] Waiting for connection context
0618 22:20:47.722987 (+ 50us) server_negotiation.cc:299] Negotiation successful
0618 22:20:47.723088 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":803,"thread_start_us":163,"threads_started":1}
I20260618 22:20:47.723285 16283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:47.715697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:33093 (local address 127.0.0.1:34398)
0618 22:20:47.716216 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:47.716230 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:20:47.716311 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:20:47.716980 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:20:47.716986 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:20:47.717024 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:20:47.717353 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:47.717362 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:47.718713 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:47.718716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:47.721362 (+ 2646us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:47.721370 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:47.722886 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:47.722890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:47.722982 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:47.723005 (+ 23us) client_negotiation.cc:769] Sending connection context
0618 22:20:47.723053 (+ 48us) client_negotiation.cc:240] Negotiation successful
0618 22:20:47.723123 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":450,"mutex_wait_us":60,"spinlock_wait_cycles":5504,"thread_start_us":119,"threads_started":1}
I20260618 22:20:47.724187 16279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:47.714439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:44555 (local address 127.0.0.1:34014)
0618 22:20:47.714710 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:47.714756 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0618 22:20:47.714869 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:20:47.718656 (+ 3787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:20:47.718661 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:20:47.718683 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:20:47.718842 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:47.718852 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:47.720897 (+ 2045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:47.720902 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:47.722391 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:47.722404 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:47.723721 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:47.723727 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:47.723859 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:47.723897 (+ 38us) client_negotiation.cc:769] Sending connection context
0618 22:20:47.723964 (+ 67us) client_negotiation.cc:240] Negotiation successful
0618 22:20:47.724050 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"mutex_wait_us":3,"thread_start_us":58,"threads_started":1}
I20260618 22:20:47.724437 16284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:47.716112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34398 (local address 127.0.98.65:33093)
0618 22:20:47.716475 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0618 22:20:47.716490 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0618 22:20:47.716513 (+ 23us) server_negotiation.cc:407] Connection header received
0618 22:20:47.716593 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:20:47.716600 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:20:47.716800 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:20:47.716962 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:20:47.717478 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:47.718583 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:47.721497 (+ 2914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:47.722755 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:47.723852 (+ 1097us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:47.723982 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0618 22:20:47.724032 (+ 50us) server_negotiation.cc:299] Negotiation successful
0618 22:20:47.724128 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":80,"threads_started":1}
I20260618 22:20:47.724824 16280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:47.714572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34014 (local address 127.0.98.67:44555)
0618 22:20:47.718158 (+ 3586us) server_negotiation.cc:206] Beginning negotiation
0618 22:20:47.718172 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0618 22:20:47.718197 (+ 25us) server_negotiation.cc:407] Connection header received
0618 22:20:47.718280 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:20:47.718288 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:20:47.718490 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:20:47.719162 (+ 672us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:20:47.719340 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:47.720727 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:47.722568 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:47.723569 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:47.724433 (+ 864us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:47.724572 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0618 22:20:47.724617 (+ 45us) server_negotiation.cc:299] Negotiation successful
0618 22:20:47.724694 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3461,"thread_start_us":193,"threads_started":1}
I20260618 22:20:47.726611 16068 tablet_service.cc:1467] Processing CreateTablet for tablet 147611d57207423683585fbc665e8bb5 (DEFAULT_TABLE table=test-table [id=b9a6fe0601c34747835651635cf8287a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:20:47.727041 15934 tablet_service.cc:1467] Processing CreateTablet for tablet 147611d57207423683585fbc665e8bb5 (DEFAULT_TABLE table=test-table [id=b9a6fe0601c34747835651635cf8287a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:20:47.727320 16202 tablet_service.cc:1467] Processing CreateTablet for tablet 147611d57207423683585fbc665e8bb5 (DEFAULT_TABLE table=test-table [id=b9a6fe0601c34747835651635cf8287a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:20:47.727815 16068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 147611d57207423683585fbc665e8bb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:20:47.728031 15934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 147611d57207423683585fbc665e8bb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:20:47.728113 16202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 147611d57207423683585fbc665e8bb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:20:47.736527 16285 tablet_bootstrap.cc:492] T 147611d57207423683585fbc665e8bb5 P 139063e1765148b181db5d13fdf5ff54: Bootstrap starting.
I20260618 22:20:47.738251 16287 tablet_bootstrap.cc:492] T 147611d57207423683585fbc665e8bb5 P 610a350e9c644e61bd41f8646ed9f7c7: Bootstrap starting.
I20260618 22:20:47.738498 16285 tablet_bootstrap.cc:654] T 147611d57207423683585fbc665e8bb5 P 139063e1765148b181db5d13fdf5ff54: Neither blocks nor log segments found. Creating new log.
I20260618 22:20:47.738680 16286 tablet_bootstrap.cc:492] T 147611d57207423683585fbc665e8bb5 P ae5c1a50e67f4aed8830a55970810e8e: Bootstrap starting.
I20260618 22:20:47.739833 16285 log.cc:824] T 147611d57207423683585fbc665e8bb5 P 139063e1765148b181db5d13fdf5ff54: Log is configured to *not* fsync() on all Append() calls
I20260618 22:20:47.741295 16287 tablet_bootstrap.cc:654] T 147611d57207423683585fbc665e8bb5 P 610a350e9c644e61bd41f8646ed9f7c7: Neither blocks nor log segments found. Creating new log.
I20260618 22:20:47.741870 16285 tablet_bootstrap.cc:492] T 147611d57207423683585fbc665e8bb5 P 139063e1765148b181db5d13fdf5ff54: No bootstrap required, opened a new log
I20260618 22:20:47.742089 16285 ts_tablet_manager.cc:1397] T 147611d57207423683585fbc665e8bb5 P 139063e1765148b181db5d13fdf5ff54: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260618 22:20:47.742268 16287 log.cc:824] T 147611d57207423683585fbc665e8bb5 P 610a350e9c644e61bd41f8646ed9f7c7: Log is configured to *not* fsync() on all Append() calls
I20260618 22:20:47.742440 16286 tablet_bootstrap.cc:654] T 147611d57207423683585fbc665e8bb5 P ae5c1a50e67f4aed8830a55970810e8e: Neither blocks nor log segments found. Creating new log.
I20260618 22:20:47.744092 16287 tablet_bootstrap.cc:492] T 147611d57207423683585fbc665e8bb5 P 610a350e9c644e61bd41f8646ed9f7c7: No bootstrap required, opened a new log
I20260618 22:20:47.744249 16287 ts_tablet_manager.cc:1397] T 147611d57207423683585fbc665e8bb5 P 610a350e9c644e61bd41f8646ed9f7c7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260618 22:20:47.744710 16286 log.cc:824] T 147611d57207423683585fbc665e8bb5 P ae5c1a50e67f4aed8830a55970810e8e: Log is configured to *not* fsync() on all Append() calls
I20260618 22:20:47.745918 16285 raft_consensus.cc:348] T 147611d57207423683585fbc665e8bb5 P 139063e1765148b181db5d13fdf5ff54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139063e1765148b181db5d13fdf5ff54" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 44555 } } peers { permanent_uuid: "610a350e9c644e61bd41f8646ed9f7c7" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39395 } } peers { permanent_uuid: "ae5c1a50e67f4aed8830a55970810e8e" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33093 } }
I20260618 22:20:47.746285 16285 raft_consensus.cc:374] T 147611d57207423683585fbc665e8bb5 P 139063e1765148b181db5d13fdf5ff54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:20:47.746385 16285 raft_consensus.cc:729] T 147611d57207423683585fbc665e8bb5 P 139063e1765148b181db5d13fdf5ff54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 139063e1765148b181db5d13fdf5ff54, State: Initialized, Role: FOLLOWER
I20260618 22:20:47.746904 16285 consensus_queue.cc:260] T 147611d57207423683585fbc665e8bb5 P 139063e1765148b181db5d13fdf5ff54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139063e1765148b181db5d13fdf5ff54" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 44555 } } peers { permanent_uuid: "610a350e9c644e61bd41f8646ed9f7c7" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39395 } } peers { permanent_uuid: "ae5c1a50e67f4aed8830a55970810e8e" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33093 } }
I20260618 22:20:47.747309 16286 tablet_bootstrap.cc:492] T 147611d57207423683585fbc665e8bb5 P ae5c1a50e67f4aed8830a55970810e8e: No bootstrap required, opened a new log
I20260618 22:20:47.747510 16286 ts_tablet_manager.cc:1397] T 147611d57207423683585fbc665e8bb5 P ae5c1a50e67f4aed8830a55970810e8e: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260618 22:20:47.747946 16285 ts_tablet_manager.cc:1428] T 147611d57207423683585fbc665e8bb5 P 139063e1765148b181db5d13fdf5ff54: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260618 22:20:47.747942 16287 raft_consensus.cc:348] T 147611d57207423683585fbc665e8bb5 P 610a350e9c644e61bd41f8646ed9f7c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139063e1765148b181db5d13fdf5ff54" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 44555 } } peers { permanent_uuid: "610a350e9c644e61bd41f8646ed9f7c7" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39395 } } peers { permanent_uuid: "ae5c1a50e67f4aed8830a55970810e8e" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33093 } }
I20260618 22:20:47.748186 16287 raft_consensus.cc:374] T 147611d57207423683585fbc665e8bb5 P 610a350e9c644e61bd41f8646ed9f7c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:20:47.748241 16287 raft_consensus.cc:729] T 147611d57207423683585fbc665e8bb5 P 610a350e9c644e61bd41f8646ed9f7c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 610a350e9c644e61bd41f8646ed9f7c7, State: Initialized, Role: FOLLOWER
I20260618 22:20:47.748286 16268 heartbeater.cc:503] Master 127.0.98.126:39001 was elected leader, sending a full tablet report...
I20260618 22:20:47.748674 16287 consensus_queue.cc:260] T 147611d57207423683585fbc665e8bb5 P 610a350e9c644e61bd41f8646ed9f7c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139063e1765148b181db5d13fdf5ff54" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 44555 } } peers { permanent_uuid: "610a350e9c644e61bd41f8646ed9f7c7" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39395 } } peers { permanent_uuid: "ae5c1a50e67f4aed8830a55970810e8e" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33093 } }
I20260618 22:20:47.749588 16287 ts_tablet_manager.cc:1428] T 147611d57207423683585fbc665e8bb5 P 610a350e9c644e61bd41f8646ed9f7c7: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260618 22:20:47.749743 16134 heartbeater.cc:503] Master 127.0.98.126:39001 was elected leader, sending a full tablet report...
I20260618 22:20:47.752331 16286 raft_consensus.cc:348] T 147611d57207423683585fbc665e8bb5 P ae5c1a50e67f4aed8830a55970810e8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139063e1765148b181db5d13fdf5ff54" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 44555 } } peers { permanent_uuid: "610a350e9c644e61bd41f8646ed9f7c7" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39395 } } peers { permanent_uuid: "ae5c1a50e67f4aed8830a55970810e8e" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33093 } }
I20260618 22:20:47.752641 16286 raft_consensus.cc:374] T 147611d57207423683585fbc665e8bb5 P ae5c1a50e67f4aed8830a55970810e8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:20:47.752748 16286 raft_consensus.cc:729] T 147611d57207423683585fbc665e8bb5 P ae5c1a50e67f4aed8830a55970810e8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae5c1a50e67f4aed8830a55970810e8e, State: Initialized, Role: FOLLOWER
I20260618 22:20:47.753224 16286 consensus_queue.cc:260] T 147611d57207423683585fbc665e8bb5 P ae5c1a50e67f4aed8830a55970810e8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139063e1765148b181db5d13fdf5ff54" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 44555 } } peers { permanent_uuid: "610a350e9c644e61bd41f8646ed9f7c7" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39395 } } peers { permanent_uuid: "ae5c1a50e67f4aed8830a55970810e8e" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33093 } }
I20260618 22:20:47.753872 16000 heartbeater.cc:503] Master 127.0.98.126:39001 was elected leader, sending a full tablet report...
I20260618 22:20:47.754221 16286 ts_tablet_manager.cc:1428] T 147611d57207423683585fbc665e8bb5 P ae5c1a50e67f4aed8830a55970810e8e: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260618 22:20:47.757982 16291 raft_consensus.cc:482] T 147611d57207423683585fbc665e8bb5 P 139063e1765148b181db5d13fdf5ff54 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:20:47.758129 16291 raft_consensus.cc:504] T 147611d57207423683585fbc665e8bb5 P 139063e1765148b181db5d13fdf5ff54 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139063e1765148b181db5d13fdf5ff54" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 44555 } } peers { permanent_uuid: "610a350e9c644e61bd41f8646ed9f7c7" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39395 } } peers { permanent_uuid: "ae5c1a50e67f4aed8830a55970810e8e" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33093 } }
I20260618 22:20:47.759063 16291 leader_election.cc:288] T 147611d57207423683585fbc665e8bb5 P 139063e1765148b181db5d13fdf5ff54 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 610a350e9c644e61bd41f8646ed9f7c7 (127.0.98.66:39395), ae5c1a50e67f4aed8830a55970810e8e (127.0.98.65:33093)
I20260618 22:20:47.763465 16270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:47.759204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:39395 (local address 127.0.98.67:48525)
0618 22:20:47.759499 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:47.759514 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:20:47.759613 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:20:47.759913 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:20:47.759916 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:20:47.759930 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:20:47.760041 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:47.760046 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:47.760996 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:47.761000 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:47.762338 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:47.762349 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:47.763056 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:47.763059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:47.763163 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:47.763183 (+ 20us) client_negotiation.cc:769] Sending connection context
0618 22:20:47.763274 (+ 91us) client_negotiation.cc:240] Negotiation successful
0618 22:20:47.763342 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":81}
I20260618 22:20:47.763464 16282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:47.759347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:48525 (local address 127.0.98.66:39395)
0618 22:20:47.759470 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0618 22:20:47.759475 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:20:47.759708 (+ 233us) server_negotiation.cc:407] Connection header received
0618 22:20:47.759754 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:20:47.759759 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:20:47.759800 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:20:47.759891 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:20:47.760166 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:47.760862 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:47.762469 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:47.762935 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:47.763043 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:47.763108 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0618 22:20:47.763290 (+ 182us) server_negotiation.cc:299] Negotiation successful
0618 22:20:47.763353 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260618 22:20:47.764052 16088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "147611d57207423683585fbc665e8bb5" candidate_uuid: "139063e1765148b181db5d13fdf5ff54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "610a350e9c644e61bd41f8646ed9f7c7" is_pre_election: true
I20260618 22:20:47.764214 16284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:47.759680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:58239 (local address 127.0.98.65:33093)
0618 22:20:47.759818 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0618 22:20:47.759824 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:20:47.759838 (+ 14us) server_negotiation.cc:407] Connection header received
0618 22:20:47.759894 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:20:47.759899 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:20:47.759950 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:20:47.760043 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:20:47.760295 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:47.761184 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:47.762687 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:47.763466 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:47.763584 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:47.763664 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0618 22:20:47.764035 (+ 371us) server_negotiation.cc:299] Negotiation successful
0618 22:20:47.764113 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260618 22:20:47.764297 16088 raft_consensus.cc:2454] T 147611d57207423683585fbc665e8bb5 P 610a350e9c644e61bd41f8646ed9f7c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 139063e1765148b181db5d13fdf5ff54 in term 0.
I20260618 22:20:47.764393 16294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:47.759324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:33093 (local address 127.0.98.67:58239)
0618 22:20:47.759619 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:47.759631 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 22:20:47.759701 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:20:47.760044 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:20:47.760046 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:20:47.760059 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:20:47.760172 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:47.760178 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:47.761299 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:47.761302 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:47.762548 (+ 1246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:20:47.762557 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:20:47.763749 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:20:47.763755 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:20:47.763891 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:47.763921 (+ 30us) client_negotiation.cc:769] Sending connection context
0618 22:20:47.764006 (+ 85us) client_negotiation.cc:240] Negotiation successful
0618 22:20:47.764091 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"mutex_wait_us":40,"thread_start_us":103,"threads_started":1}
I20260618 22:20:47.764748 16157 leader_election.cc:302] T 147611d57207423683585fbc665e8bb5 P 139063e1765148b181db5d13fdf5ff54 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 139063e1765148b181db5d13fdf5ff54, 610a350e9c644e61bd41f8646ed9f7c7; no voters:
I20260618 22:20:47.764935 15954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "147611d57207423683585fbc665e8bb5" candidate_uuid: "139063e1765148b181db5d13fdf5ff54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae5c1a50e67f4aed8830a55970810e8e" is_pre_election: true
I20260618 22:20:47.764981 16291 raft_consensus.cc:2781] T 147611d57207423683585fbc665e8bb5 P 139063e1765148b181db5d13fdf5ff54 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:20:47.765054 16291 raft_consensus.cc:482] T 147611d57207423683585fbc665e8bb5 P 139063e1765148b181db5d13fdf5ff54 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:20:47.765117 16291 raft_consensus.cc:3037] T 147611d57207423683585fbc665e8bb5 P 139063e1765148b181db5d13fdf5ff54 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:20:47.765200 15954 raft_consensus.cc:2454] T 147611d57207423683585fbc665e8bb5 P ae5c1a50e67f4aed8830a55970810e8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 139063e1765148b181db5d13fdf5ff54 in term 0.
I20260618 22:20:47.766223 16291 raft_consensus.cc:504] T 147611d57207423683585fbc665e8bb5 P 139063e1765148b181db5d13fdf5ff54 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139063e1765148b181db5d13fdf5ff54" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 44555 } } peers { permanent_uuid: "610a350e9c644e61bd41f8646ed9f7c7" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39395 } } peers { permanent_uuid: "ae5c1a50e67f4aed8830a55970810e8e" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33093 } }
I20260618 22:20:47.766563 16291 leader_election.cc:288] T 147611d57207423683585fbc665e8bb5 P 139063e1765148b181db5d13fdf5ff54 [CANDIDATE]: Term 1 election: Requested vote from peers 610a350e9c644e61bd41f8646ed9f7c7 (127.0.98.66:39395), ae5c1a50e67f4aed8830a55970810e8e (127.0.98.65:33093)
I20260618 22:20:47.766810 16088 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "147611d57207423683585fbc665e8bb5" candidate_uuid: "139063e1765148b181db5d13fdf5ff54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "610a350e9c644e61bd41f8646ed9f7c7"
I20260618 22:20:47.766934 16088 raft_consensus.cc:3037] T 147611d57207423683585fbc665e8bb5 P 610a350e9c644e61bd41f8646ed9f7c7 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:20:47.766989 15954 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "147611d57207423683585fbc665e8bb5" candidate_uuid: "139063e1765148b181db5d13fdf5ff54" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae5c1a50e67f4aed8830a55970810e8e"
I20260618 22:20:47.767082 15954 raft_consensus.cc:3037] T 147611d57207423683585fbc665e8bb5 P ae5c1a50e67f4aed8830a55970810e8e [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:20:47.768147 15954 raft_consensus.cc:2454] T 147611d57207423683585fbc665e8bb5 P ae5c1a50e67f4aed8830a55970810e8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 139063e1765148b181db5d13fdf5ff54 in term 1.
I20260618 22:20:47.768146 16088 raft_consensus.cc:2454] T 147611d57207423683585fbc665e8bb5 P 610a350e9c644e61bd41f8646ed9f7c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 139063e1765148b181db5d13fdf5ff54 in term 1.
I20260618 22:20:47.768504 16157 leader_election.cc:302] T 147611d57207423683585fbc665e8bb5 P 139063e1765148b181db5d13fdf5ff54 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 139063e1765148b181db5d13fdf5ff54, 610a350e9c644e61bd41f8646ed9f7c7; no voters:
I20260618 22:20:47.768728 16291 raft_consensus.cc:2781] T 147611d57207423683585fbc665e8bb5 P 139063e1765148b181db5d13fdf5ff54 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:20:47.768922 16291 raft_consensus.cc:686] T 147611d57207423683585fbc665e8bb5 P 139063e1765148b181db5d13fdf5ff54 [term 1 LEADER]: Becoming Leader. State: Replica: 139063e1765148b181db5d13fdf5ff54, State: Running, Role: LEADER
I20260618 22:20:47.769215 16291 consensus_queue.cc:237] T 147611d57207423683585fbc665e8bb5 P 139063e1765148b181db5d13fdf5ff54 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139063e1765148b181db5d13fdf5ff54" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 44555 } } peers { permanent_uuid: "610a350e9c644e61bd41f8646ed9f7c7" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39395 } } peers { permanent_uuid: "ae5c1a50e67f4aed8830a55970810e8e" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33093 } }
I20260618 22:20:47.772039 15808 catalog_manager.cc:5495] T 147611d57207423683585fbc665e8bb5 P 139063e1765148b181db5d13fdf5ff54 reported cstate change: term changed from 0 to 1, leader changed from <none> to 139063e1765148b181db5d13fdf5ff54 (127.0.98.67). New cstate: current_term: 1 leader_uuid: "139063e1765148b181db5d13fdf5ff54" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139063e1765148b181db5d13fdf5ff54" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 44555 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "610a350e9c644e61bd41f8646ed9f7c7" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae5c1a50e67f4aed8830a55970810e8e" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33093 } health_report { overall_health: UNKNOWN } } }
I20260618 22:20:47.792778 16280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:47.788322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34024 (local address 127.0.98.67:44555)
0618 22:20:47.788442 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0618 22:20:47.788446 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:20:47.788506 (+ 60us) server_negotiation.cc:407] Connection header received
0618 22:20:47.788752 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:20:47.788757 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:20:47.788851 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:20:47.788984 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:20:47.789182 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:47.790119 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:47.790936 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:20:47.791389 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:20:47.791518 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:20:47.791768 (+ 250us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:20:47.792336 (+ 568us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:20:47.792474 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0618 22:20:47.792512 (+ 38us) server_negotiation.cc:299] Negotiation successful
0618 22:20:47.792582 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260618 22:20:47.798573 15954 raft_consensus.cc:1261] T 147611d57207423683585fbc665e8bb5 P ae5c1a50e67f4aed8830a55970810e8e [term 1 FOLLOWER]: Refusing update from remote peer 139063e1765148b181db5d13fdf5ff54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:20:47.798573 16088 raft_consensus.cc:1261] T 147611d57207423683585fbc665e8bb5 P 610a350e9c644e61bd41f8646ed9f7c7 [term 1 FOLLOWER]: Refusing update from remote peer 139063e1765148b181db5d13fdf5ff54: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:20:47.799319 16291 consensus_queue.cc:1035] T 147611d57207423683585fbc665e8bb5 P 139063e1765148b181db5d13fdf5ff54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "610a350e9c644e61bd41f8646ed9f7c7" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:20:47.799666 16295 consensus_queue.cc:1035] T 147611d57207423683585fbc665e8bb5 P 139063e1765148b181db5d13fdf5ff54 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae5c1a50e67f4aed8830a55970810e8e" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:20:47.809335 16298 mvcc.cc:205] Tried to move back new op lower bound from 7298339830975848448 to 7298339830865240064. Current Snapshot: MvccSnapshot[applied={T|T < 7298339830975848448}]
I20260618 22:20:47.810995 16301 mvcc.cc:205] Tried to move back new op lower bound from 7298339830975848448 to 7298339830865240064. Current Snapshot: MvccSnapshot[applied={T|T < 7298339830975848448}]
I20260618 22:20:47.817237 15809 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:20:47.817405 15809 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43370:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:20:47.820111 15809 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 56d1566a923f47dca11d653707ba497a: Sending DeleteTablet for 3 replicas of tablet 147611d57207423683585fbc665e8bb5
I20260618 22:20:47.820864 16202 tablet_service.cc:1514] Processing DeleteTablet for tablet 147611d57207423683585fbc665e8bb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:20:47 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:34014
I20260618 22:20:47.821095 15934 tablet_service.cc:1514] Processing DeleteTablet for tablet 147611d57207423683585fbc665e8bb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:20:47 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:34398
I20260618 22:20:47.821573 16307 tablet_replica.cc:330] T 147611d57207423683585fbc665e8bb5 P 139063e1765148b181db5d13fdf5ff54: stopping tablet replica
I20260618 22:20:47.821676 16068 tablet_service.cc:1514] Processing DeleteTablet for tablet 147611d57207423683585fbc665e8bb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:20:47 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:42926
I20260618 22:20:47.821985 16307 raft_consensus.cc:2229] T 147611d57207423683585fbc665e8bb5 P 139063e1765148b181db5d13fdf5ff54 [term 1 LEADER]: Raft consensus shutting down.
I20260618 22:20:47.822325 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 15871
I20260618 22:20:47.822438 16307 raft_consensus.cc:2258] T 147611d57207423683585fbc665e8bb5 P 139063e1765148b181db5d13fdf5ff54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:20:47.824018 16307 ts_tablet_manager.cc:1905] T 147611d57207423683585fbc665e8bb5 P 139063e1765148b181db5d13fdf5ff54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:20:47.825985 16309 tablet_replica.cc:330] T 147611d57207423683585fbc665e8bb5 P 610a350e9c644e61bd41f8646ed9f7c7: stopping tablet replica
I20260618 22:20:47.827401 16307 ts_tablet_manager.cc:1918] T 147611d57207423683585fbc665e8bb5 P 139063e1765148b181db5d13fdf5ff54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:20:47.827584 16307 log.cc:1196] T 147611d57207423683585fbc665e8bb5 P 139063e1765148b181db5d13fdf5ff54: Deleting WAL directory at /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-2/wal/wals/147611d57207423683585fbc665e8bb5
I20260618 22:20:47.828013 16307 ts_tablet_manager.cc:1939] T 147611d57207423683585fbc665e8bb5 P 139063e1765148b181db5d13fdf5ff54: Deleting consensus metadata
I20260618 22:20:47.829303 15793 catalog_manager.cc:4841] TS 139063e1765148b181db5d13fdf5ff54 (127.0.98.67:44555): tablet 147611d57207423683585fbc665e8bb5 (table test-table [id=b9a6fe0601c34747835651635cf8287a]) successfully deleted
I20260618 22:20:47.830250 16309 raft_consensus.cc:2229] T 147611d57207423683585fbc665e8bb5 P 610a350e9c644e61bd41f8646ed9f7c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:20:47.830729 16309 raft_consensus.cc:2258] T 147611d57207423683585fbc665e8bb5 P 610a350e9c644e61bd41f8646ed9f7c7 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260618 22:20:47.831800 15794 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.65:33093 (error 108)
I20260618 22:20:47.832469 16309 ts_tablet_manager.cc:1905] T 147611d57207423683585fbc665e8bb5 P 610a350e9c644e61bd41f8646ed9f7c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:20:47.833052 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 16005
I20260618 22:20:47.833124 16279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:47.832842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:33093 (local address 127.0.0.1:34408)
0618 22:20:47.832976 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:47.833048 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.65:33093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260618 22:20:47.834494 15794 catalog_manager.cc:4584] TS ae5c1a50e67f4aed8830a55970810e8e (127.0.98.65:33093): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 147611d57207423683585fbc665e8bb5: Network error: Client connection negotiation failed: client connection to 127.0.98.65:33093: connect: Connection refused (error 111)
I20260618 22:20:47.839167 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 16139
I20260618 22:20:47.839963 16279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:20:47.839348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:39395 (local address 127.0.0.1:42934)
0618 22:20:47.839833 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0618 22:20:47.839891 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.66:39395: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397}
I20260618 22:20:47.845028 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 15777
I20260618 22:21:07.851111 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:39001
--webserver_interface=127.0.98.126
--webserver_port=37291
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:38325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:39001
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:21:07.958455 16311 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:21:07.958702 16311 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:21:07.958743 16311 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:21:07.962620 16311 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260618 22:21:07.962657 16311 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260618 22:21:07.962672 16311 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260618 22:21:07.962685 16311 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:21:07.962703 16311 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:21:07.962720 16311 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:21:07.962736 16311 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:21:07.962750 16311 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:21:07.967820 16311 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:38325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:39001
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:39001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=37291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16311
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:21:07.968719 16311 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:21:07.969653 16311 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:21:07.982158 16320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:21:07.982210 16311 server_base.cc:1034] running on GCE node
W20260618 22:21:07.982159 16318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:21:07.982159 16317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:21:07.983121 16311 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:21:07.984205 16311 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:21:07.985396 16311 hybrid_clock.cc:648] HybridClock initialized: now 1781821267985380 us; error 38 us; skew 500 ppm
Jun 18 22:21:07 dist-test-slave-g6r4 krb5kdc[15764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821267, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:21:07.990870 16311 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260618 22:21:07.991739 16311 webserver.cc:460] Webserver started at http://127.0.98.126:37291/ using document root <none> and password file <none>
I20260618 22:21:07.992265 16311 fs_manager.cc:359] Metadata directory not provided
I20260618 22:21:07.992333 16311 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:21:07.996033 16311 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260618 22:21:07.998631 16327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:08.000051 16311 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:21:08.000187 16311 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/master-0/wal
uuid: "56d1566a923f47dca11d653707ba497a"
format_stamp: "Formatted at 2026-06-18 22:20:46 on dist-test-slave-g6r4"
server_key: "339cc53340c9db76e65087b522b03ba8"
server_key_iv: "9a85caacb642db088086e741d0d79546"
server_key_version: "encryptionkey@0"
I20260618 22:21:08.000777 16311 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:21:08.024137 16311 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:21:08.027138 16311 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:21:08.027304 16311 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:21:08.034817 16311 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:39001
I20260618 22:21:08.035094 16379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:39001 every 8 connection(s)
I20260618 22:21:08.035954 16311 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
I20260618 22:21:08.037386 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 16311
I20260618 22:21:08.037886 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:33093
--local_ip_for_outbound_sockets=127.0.98.65
--tserver_master_addrs=127.0.98.126:39001
--webserver_port=35901
--webserver_interface=127.0.98.65
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:38325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 22:21:08.040899 16380 sys_catalog.cc:263] Verifying existing consensus state
I20260618 22:21:08.042572 16380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56d1566a923f47dca11d653707ba497a: Bootstrap starting.
I20260618 22:21:08.058650 16380 log.cc:824] T 00000000000000000000000000000000 P 56d1566a923f47dca11d653707ba497a: Log is configured to *not* fsync() on all Append() calls
I20260618 22:21:08.066990 16380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56d1566a923f47dca11d653707ba497a: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260618 22:21:08.067354 16380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56d1566a923f47dca11d653707ba497a: Bootstrap complete.
I20260618 22:21:08.071251 16380 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56d1566a923f47dca11d653707ba497a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d1566a923f47dca11d653707ba497a" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39001 } }
I20260618 22:21:08.071612 16380 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56d1566a923f47dca11d653707ba497a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56d1566a923f47dca11d653707ba497a, State: Initialized, Role: FOLLOWER
I20260618 22:21:08.072178 16380 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56d1566a923f47dca11d653707ba497a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d1566a923f47dca11d653707ba497a" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39001 } }
I20260618 22:21:08.072293 16380 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56d1566a923f47dca11d653707ba497a [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:21:08.072381 16380 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56d1566a923f47dca11d653707ba497a [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:21:08.072482 16380 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56d1566a923f47dca11d653707ba497a [term 1 FOLLOWER]: Advancing to term 2
I20260618 22:21:08.074347 16380 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56d1566a923f47dca11d653707ba497a [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d1566a923f47dca11d653707ba497a" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39001 } }
I20260618 22:21:08.074709 16380 leader_election.cc:302] T 00000000000000000000000000000000 P 56d1566a923f47dca11d653707ba497a [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56d1566a923f47dca11d653707ba497a; no voters:
I20260618 22:21:08.075016 16380 leader_election.cc:288] T 00000000000000000000000000000000 P 56d1566a923f47dca11d653707ba497a [CANDIDATE]: Term 2 election: Requested vote from peers
I20260618 22:21:08.075156 16384 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56d1566a923f47dca11d653707ba497a [term 2 FOLLOWER]: Leader election won for term 2
I20260618 22:21:08.075402 16384 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56d1566a923f47dca11d653707ba497a [term 2 LEADER]: Becoming Leader. State: Replica: 56d1566a923f47dca11d653707ba497a, State: Running, Role: LEADER
I20260618 22:21:08.075742 16384 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56d1566a923f47dca11d653707ba497a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d1566a923f47dca11d653707ba497a" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39001 } }
I20260618 22:21:08.076143 16380 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56d1566a923f47dca11d653707ba497a [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:21:08.076679 16384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56d1566a923f47dca11d653707ba497a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56d1566a923f47dca11d653707ba497a. Latest consensus state: current_term: 2 leader_uuid: "56d1566a923f47dca11d653707ba497a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d1566a923f47dca11d653707ba497a" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39001 } } }
I20260618 22:21:08.076871 16384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56d1566a923f47dca11d653707ba497a [sys.catalog]: This master's current role is: LEADER
I20260618 22:21:08.077131 16385 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56d1566a923f47dca11d653707ba497a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "56d1566a923f47dca11d653707ba497a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d1566a923f47dca11d653707ba497a" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39001 } } }
I20260618 22:21:08.077247 16385 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56d1566a923f47dca11d653707ba497a [sys.catalog]: This master's current role is: LEADER
I20260618 22:21:08.077350 16390 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:21:08.081827 16390 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:21:08.082520 16390 catalog_manager.cc:1259] Loaded cluster ID: 1146874a6ad643a9a163dfb9a1560f69
I20260618 22:21:08.082584 16390 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:21:08.085130 16390 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:21:08.085840 16390 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 56d1566a923f47dca11d653707ba497a: Loaded TSK: 0
I20260618 22:21:08.102213 16390 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 56d1566a923f47dca11d653707ba497a: Generated new TSK 1
I20260618 22:21:08.102455 16390 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260618 22:21:08.155870 16382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:21:08.156141 16382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:21:08.156214 16382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:21:08.160181 16382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:21:08.160259 16382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:21:08.160346 16382 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260618 22:21:08.165375 16382 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:38325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:33093
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=35901
--tserver_master_addrs=127.0.98.126:39001
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16382
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:21:08.166333 16382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:21:08.167320 16382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:21:08.176350 16407 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:21:08.176431 16408 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:21:08.176627 16410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:21:08.176854 16382 server_base.cc:1034] running on GCE node
I20260618 22:21:08.177251 16382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:21:08.177943 16382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:21:08.179162 16382 hybrid_clock.cc:648] HybridClock initialized: now 1781821268179143 us; error 36 us; skew 500 ppm
Jun 18 22:21:08 dist-test-slave-g6r4 krb5kdc[15764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821268, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:21:08.181823 16382 init.cc:377] Logged in from keytab as kudu/127.0.98.65@KRBTEST.COM (short username kudu)
I20260618 22:21:08.182770 16382 webserver.cc:460] Webserver started at http://127.0.98.65:35901/ using document root <none> and password file <none>
I20260618 22:21:08.183292 16382 fs_manager.cc:359] Metadata directory not provided
I20260618 22:21:08.183362 16382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:21:08.187016 16382 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260618 22:21:08.189201 16417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:08.190475 16382 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:21:08.190563 16382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-0/wal
uuid: "ae5c1a50e67f4aed8830a55970810e8e"
format_stamp: "Formatted at 2026-06-18 22:20:47 on dist-test-slave-g6r4"
server_key: "0eea59f8f5fe730b371cff829746fe6f"
server_key_iv: "681f3e6a2126ea0e2d2a237f13dc42d0"
server_key_version: "encryptionkey@0"
I20260618 22:21:08.191113 16382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:21:08.202414 16382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:21:08.205605 16382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:21:08.205796 16382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:21:08.206419 16382 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:21:08.207530 16424 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260618 22:21:08.211440 16382 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260618 22:21:08.211498 16382 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.001s sys 0.000s
I20260618 22:21:08.211542 16382 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260618 22:21:08.213184 16382 ts_tablet_manager.cc:610] Registered 1 tablets
I20260618 22:21:08.213238 16382 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260618 22:21:08.213486 16424 tablet_bootstrap.cc:492] T 147611d57207423683585fbc665e8bb5 P ae5c1a50e67f4aed8830a55970810e8e: Bootstrap starting.
I20260618 22:21:08.224063 16382 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:33093
I20260618 22:21:08.224262 16531 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:33093 every 8 connection(s)
I20260618 22:21:08.225258 16382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
I20260618 22:21:08.233412 16424 log.cc:824] T 147611d57207423683585fbc665e8bb5 P ae5c1a50e67f4aed8830a55970810e8e: Log is configured to *not* fsync() on all Append() calls
I20260618 22:21:08.234470 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 16382
I20260618 22:21:08.234999 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:39395
--local_ip_for_outbound_sockets=127.0.98.66
--tserver_master_addrs=127.0.98.126:39001
--webserver_port=39183
--webserver_interface=127.0.98.66
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:38325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 18 22:21:08 dist-test-slave-g6r4 krb5kdc[15764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821268, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:21:08.241809 16534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:08.227342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:60753 (local address 127.0.98.126:39001)
0618 22:21:08.227705 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:08.227718 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:08.228122 (+ 404us) server_negotiation.cc:407] Connection header received
0618 22:21:08.229196 (+ 1074us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:08.229206 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:08.229480 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:08.229792 (+ 312us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:21:08.230936 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:08.231890 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:08.233034 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:08.233504 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:08.233670 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:08.236751 (+ 3081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:21:08.236757 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:21:08.236771 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:21:08.236810 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:21:08.238732 (+ 1922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:21:08.239239 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:21:08.239245 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:21:08.239252 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:21:08.239366 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:21:08.239745 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:21:08.239748 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:21:08.239750 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:21:08.240073 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:21:08.240153 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:08.240727 (+ 574us) server_negotiation.cc:299] Negotiation successful
0618 22:21:08.240982 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":92,"threads_started":1}
I20260618 22:21:08.241822 16535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:08.227346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39001 (local address 127.0.98.65:60753)
0618 22:21:08.227949 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:08.227989 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0618 22:21:08.228802 (+ 813us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:21:08.229852 (+ 1050us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:21:08.229859 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:21:08.230307 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:21:08.230762 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:08.230774 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:08.232043 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:08.232051 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:08.232901 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:08.232910 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:08.233640 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:08.233644 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:08.233834 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:08.234557 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:21:08.234579 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:21:08.236517 (+ 1938us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:21:08.238854 (+ 2337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:21:08.238860 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:21:08.238872 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:21:08.239125 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:21:08.239464 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:21:08.239469 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:21:08.239471 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:21:08.239609 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:21:08.240221 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:21:08.240225 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:21:08.240430 (+ 205us) client_negotiation.cc:769] Sending connection context
0618 22:21:08.240616 (+ 186us) client_negotiation.cc:240] Negotiation successful
0618 22:21:08.240962 (+ 346us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":147,"threads_started":1}
I20260618 22:21:08.242537 16424 tablet_bootstrap.cc:492] T 147611d57207423683585fbc665e8bb5 P ae5c1a50e67f4aed8830a55970810e8e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260618 22:21:08.243072 16424 tablet_bootstrap.cc:492] T 147611d57207423683585fbc665e8bb5 P ae5c1a50e67f4aed8830a55970810e8e: Bootstrap complete.
I20260618 22:21:08.243937 16532 heartbeater.cc:347] Connected to a master server at 127.0.98.126:39001
I20260618 22:21:08.244159 16532 heartbeater.cc:464] Registering TS with master...
I20260618 22:21:08.244560 16424 ts_tablet_manager.cc:1397] T 147611d57207423683585fbc665e8bb5 P ae5c1a50e67f4aed8830a55970810e8e: Time spent bootstrapping tablet: real 0.031s user 0.024s sys 0.004s
I20260618 22:21:08.244611 16532 heartbeater.cc:511] Master 127.0.98.126:39001 requested a full tablet report, sending...
I20260618 22:21:08.247087 16343 ts_manager.cc:194] Registered new tserver with Master: ae5c1a50e67f4aed8830a55970810e8e (127.0.98.65:33093)
I20260618 22:21:08.248245 16424 raft_consensus.cc:348] T 147611d57207423683585fbc665e8bb5 P ae5c1a50e67f4aed8830a55970810e8e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139063e1765148b181db5d13fdf5ff54" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 44555 } } peers { permanent_uuid: "610a350e9c644e61bd41f8646ed9f7c7" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39395 } } peers { permanent_uuid: "ae5c1a50e67f4aed8830a55970810e8e" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33093 } }
I20260618 22:21:08.248726 16424 raft_consensus.cc:729] T 147611d57207423683585fbc665e8bb5 P ae5c1a50e67f4aed8830a55970810e8e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae5c1a50e67f4aed8830a55970810e8e, State: Initialized, Role: FOLLOWER
I20260618 22:21:08.249187 16424 consensus_queue.cc:260] T 147611d57207423683585fbc665e8bb5 P ae5c1a50e67f4aed8830a55970810e8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139063e1765148b181db5d13fdf5ff54" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 44555 } } peers { permanent_uuid: "610a350e9c644e61bd41f8646ed9f7c7" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39395 } } peers { permanent_uuid: "ae5c1a50e67f4aed8830a55970810e8e" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33093 } }
I20260618 22:21:08.250120 16424 ts_tablet_manager.cc:1428] T 147611d57207423683585fbc665e8bb5 P ae5c1a50e67f4aed8830a55970810e8e: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260618 22:21:08.250445 16343 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.65@KRBTEST.COM'} at 127.0.98.65:60753
I20260618 22:21:08.252964 16532 heartbeater.cc:503] Master 127.0.98.126:39001 was elected leader, sending a full tablet report...
Jun 18 22:21:08 dist-test-slave-g6r4 krb5kdc[15764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821267, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for kudu/127.0.98.65@KRBTEST.COM
I20260618 22:21:08.260133 16539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:08.249754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:33093 (local address 127.0.0.1:39160)
0618 22:21:08.250832 (+ 1078us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:08.250881 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0618 22:21:08.251017 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:21:08.251483 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:21:08.251489 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:21:08.251818 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:21:08.251907 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:08.251913 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:08.252821 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:08.252826 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:08.253642 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:08.253651 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:08.254515 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:08.254519 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:08.254633 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:08.255180 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:21:08.255199 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:21:08.256643 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:21:08.258640 (+ 1997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:21:08.258647 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:21:08.258651 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:21:08.258861 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:21:08.259151 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:21:08.259153 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:21:08.259155 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:21:08.259227 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:21:08.259687 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:21:08.259691 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:21:08.259775 (+ 84us) client_negotiation.cc:769] Sending connection context
0618 22:21:08.259876 (+ 101us) client_negotiation.cc:240] Negotiation successful
0618 22:21:08.260005 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":913,"thread_start_us":137,"threads_started":1}
I20260618 22:21:08.260314 16540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:08.250504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39160 (local address 127.0.98.65:33093)
0618 22:21:08.250885 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:08.250896 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:08.250958 (+ 62us) server_negotiation.cc:407] Connection header received
0618 22:21:08.251166 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:08.251172 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:08.251352 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:08.251483 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:21:08.252029 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:08.252710 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:08.253758 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:08.254297 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:08.254397 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:08.256769 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:21:08.256779 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:21:08.256791 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:21:08.256815 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:21:08.258538 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:21:08.258963 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:21:08.258969 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:21:08.258971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:21:08.259063 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:21:08.259309 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:21:08.259312 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:21:08.259314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:21:08.259580 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:21:08.259699 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:08.259985 (+ 286us) server_negotiation.cc:299] Negotiation successful
0618 22:21:08.260144 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":84,"threads_started":1}
I20260618 22:21:08.261118 16466 tablet_service.cc:1514] Processing DeleteTablet for tablet 147611d57207423683585fbc665e8bb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:20:47 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:39160
I20260618 22:21:08.261958 16541 tablet_replica.cc:330] stopping tablet replica
I20260618 22:21:08.262245 16541 raft_consensus.cc:2229] T 147611d57207423683585fbc665e8bb5 P ae5c1a50e67f4aed8830a55970810e8e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:21:08.262481 16541 raft_consensus.cc:2258] T 147611d57207423683585fbc665e8bb5 P ae5c1a50e67f4aed8830a55970810e8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:21:08.263471 16541 ts_tablet_manager.cc:1905] T 147611d57207423683585fbc665e8bb5 P ae5c1a50e67f4aed8830a55970810e8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:21:08.267751 16541 ts_tablet_manager.cc:1918] T 147611d57207423683585fbc665e8bb5 P ae5c1a50e67f4aed8830a55970810e8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:21:08.267846 16541 log.cc:1196] T 147611d57207423683585fbc665e8bb5 P ae5c1a50e67f4aed8830a55970810e8e: Deleting WAL directory at /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-0/wal/wals/147611d57207423683585fbc665e8bb5
I20260618 22:21:08.268111 16541 ts_tablet_manager.cc:1939] T 147611d57207423683585fbc665e8bb5 P ae5c1a50e67f4aed8830a55970810e8e: Deleting consensus metadata
I20260618 22:21:08.269887 16329 catalog_manager.cc:4841] TS ae5c1a50e67f4aed8830a55970810e8e (127.0.98.65:33093): tablet 147611d57207423683585fbc665e8bb5 (table test-table [id=b9a6fe0601c34747835651635cf8287a]) successfully deleted
W20260618 22:21:08.351603 16537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:21:08.351823 16537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:21:08.351863 16537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:21:08.355852 16537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:21:08.355906 16537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:21:08.355994 16537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260618 22:21:08.360929 16537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:38325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:39395
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=39183
--tserver_master_addrs=127.0.98.126:39001
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:21:08.361877 16537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:21:08.362928 16537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260618 22:21:08.372073 16537 server_base.cc:1034] running on GCE node
W20260618 22:21:08.372140 16546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:21:08.372126 16547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:21:08.372239 16549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:21:08.372668 16537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:21:08.373288 16537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:21:08.374464 16537 hybrid_clock.cc:648] HybridClock initialized: now 1781821268374430 us; error 48 us; skew 500 ppm
Jun 18 22:21:08 dist-test-slave-g6r4 krb5kdc[15764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821268, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:21:08.377372 16537 init.cc:377] Logged in from keytab as kudu/127.0.98.66@KRBTEST.COM (short username kudu)
I20260618 22:21:08.378249 16537 webserver.cc:460] Webserver started at http://127.0.98.66:39183/ using document root <none> and password file <none>
I20260618 22:21:08.378763 16537 fs_manager.cc:359] Metadata directory not provided
I20260618 22:21:08.378829 16537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:21:08.382340 16537 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:21:08.384361 16556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:08.385456 16537 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:21:08.385586 16537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-1/wal
uuid: "610a350e9c644e61bd41f8646ed9f7c7"
format_stamp: "Formatted at 2026-06-18 22:20:47 on dist-test-slave-g6r4"
server_key: "be8fc677a8c3224a0f362f5a111672d8"
server_key_iv: "9375ad4ed12a38ad90323ab04f36043b"
server_key_version: "encryptionkey@0"
I20260618 22:21:08.386130 16537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:21:08.402140 16537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:21:08.405339 16537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:21:08.405571 16537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:21:08.406247 16537 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:21:08.407323 16563 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260618 22:21:08.411268 16537 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260618 22:21:08.411327 16537 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260618 22:21:08.411398 16537 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260618 22:21:08.412974 16537 ts_tablet_manager.cc:610] Registered 1 tablets
I20260618 22:21:08.413020 16537 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260618 22:21:08.413285 16563 tablet_bootstrap.cc:492] T 147611d57207423683585fbc665e8bb5 P 610a350e9c644e61bd41f8646ed9f7c7: Bootstrap starting.
I20260618 22:21:08.419309 16563 log.cc:824] T 147611d57207423683585fbc665e8bb5 P 610a350e9c644e61bd41f8646ed9f7c7: Log is configured to *not* fsync() on all Append() calls
I20260618 22:21:08.422785 16537 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:39395
I20260618 22:21:08.422855 16671 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:39395 every 8 connection(s)
I20260618 22:21:08.423636 16537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
I20260618 22:21:08.427150 16563 tablet_bootstrap.cc:492] T 147611d57207423683585fbc665e8bb5 P 610a350e9c644e61bd41f8646ed9f7c7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260618 22:21:08.427573 16563 tablet_bootstrap.cc:492] T 147611d57207423683585fbc665e8bb5 P 610a350e9c644e61bd41f8646ed9f7c7: Bootstrap complete.
I20260618 22:21:08.427922 16563 ts_tablet_manager.cc:1397] T 147611d57207423683585fbc665e8bb5 P 610a350e9c644e61bd41f8646ed9f7c7: Time spent bootstrapping tablet: real 0.015s user 0.012s sys 0.000s
I20260618 22:21:08.430742 16563 raft_consensus.cc:348] T 147611d57207423683585fbc665e8bb5 P 610a350e9c644e61bd41f8646ed9f7c7 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139063e1765148b181db5d13fdf5ff54" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 44555 } } peers { permanent_uuid: "610a350e9c644e61bd41f8646ed9f7c7" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39395 } } peers { permanent_uuid: "ae5c1a50e67f4aed8830a55970810e8e" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33093 } }
I20260618 22:21:08.431208 16563 raft_consensus.cc:729] T 147611d57207423683585fbc665e8bb5 P 610a350e9c644e61bd41f8646ed9f7c7 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 610a350e9c644e61bd41f8646ed9f7c7, State: Initialized, Role: FOLLOWER
I20260618 22:21:08.431723 16563 consensus_queue.cc:260] T 147611d57207423683585fbc665e8bb5 P 610a350e9c644e61bd41f8646ed9f7c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139063e1765148b181db5d13fdf5ff54" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 44555 } } peers { permanent_uuid: "610a350e9c644e61bd41f8646ed9f7c7" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39395 } } peers { permanent_uuid: "ae5c1a50e67f4aed8830a55970810e8e" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33093 } }
I20260618 22:21:08.432723 16563 ts_tablet_manager.cc:1428] T 147611d57207423683585fbc665e8bb5 P 610a350e9c644e61bd41f8646ed9f7c7: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260618 22:21:08.433284 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 16537
Jun 18 22:21:08 dist-test-slave-g6r4 krb5kdc[15764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821268, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:21:08.433764 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:44555
--local_ip_for_outbound_sockets=127.0.98.67
--tserver_master_addrs=127.0.98.126:39001
--webserver_port=39675
--webserver_interface=127.0.98.67
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:38325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 22:21:08.437852 16534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:08.425457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:34895 (local address 127.0.98.126:39001)
0618 22:21:08.425590 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:08.425594 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:08.426198 (+ 604us) server_negotiation.cc:407] Connection header received
0618 22:21:08.427117 (+ 919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:08.427121 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:08.427184 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:08.427263 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:21:08.428735 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:08.429260 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:08.430855 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:08.431092 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:08.431172 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:08.434236 (+ 3064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:21:08.434241 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:21:08.434249 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:21:08.434292 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:21:08.435882 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:21:08.436396 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:21:08.436400 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:21:08.436402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:21:08.436479 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:21:08.436820 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:21:08.436823 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:21:08.436825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:21:08.437001 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:21:08.437102 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:08.437561 (+ 459us) server_negotiation.cc:299] Negotiation successful
0618 22:21:08.437688 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260618 22:21:08.438580 16674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:08.425575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39001 (local address 127.0.98.66:34895)
0618 22:21:08.426055 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:08.426096 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0618 22:21:08.426883 (+ 787us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:21:08.427489 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:21:08.427501 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:21:08.427966 (+ 465us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:21:08.428582 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:08.428601 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:08.429405 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:08.429409 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:08.430738 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:08.430750 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:08.431206 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:08.431210 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:08.431402 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:08.432340 (+ 938us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:21:08.432359 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:21:08.434049 (+ 1690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:21:08.436022 (+ 1973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:21:08.436027 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:21:08.436040 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:21:08.436288 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:21:08.436582 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:21:08.436585 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:21:08.436587 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:21:08.436719 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:21:08.437100 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:21:08.437106 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:21:08.437331 (+ 225us) client_negotiation.cc:769] Sending connection context
0618 22:21:08.437498 (+ 167us) client_negotiation.cc:240] Negotiation successful
0618 22:21:08.437796 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":109,"threads_started":1}
I20260618 22:21:08.439810 16672 heartbeater.cc:347] Connected to a master server at 127.0.98.126:39001
I20260618 22:21:08.440078 16672 heartbeater.cc:464] Registering TS with master...
I20260618 22:21:08.440676 16672 heartbeater.cc:511] Master 127.0.98.126:39001 requested a full tablet report, sending...
I20260618 22:21:08.442787 16343 ts_manager.cc:194] Registered new tserver with Master: 610a350e9c644e61bd41f8646ed9f7c7 (127.0.98.66:39395)
I20260618 22:21:08.444330 16343 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.66@KRBTEST.COM'} at 127.0.98.66:34895
I20260618 22:21:08.446897 16672 heartbeater.cc:503] Master 127.0.98.126:39001 was elected leader, sending a full tablet report...
Jun 18 22:21:08 dist-test-slave-g6r4 krb5kdc[15764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821267, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for kudu/127.0.98.66@KRBTEST.COM
I20260618 22:21:08.452683 16539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:08.443891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:39395 (local address 127.0.0.1:45686)
0618 22:21:08.444117 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:08.444132 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:21:08.444209 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:21:08.444825 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:21:08.444828 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:21:08.445026 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:21:08.445102 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:08.445108 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:08.446177 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:08.446182 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:08.446704 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:08.446711 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:08.447413 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:08.447418 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:08.447527 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:08.448085 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:21:08.448100 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:21:08.449337 (+ 1237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:21:08.451171 (+ 1834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:21:08.451176 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:21:08.451178 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:21:08.451366 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:21:08.451653 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:21:08.451657 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:21:08.451659 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:21:08.451744 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:21:08.452169 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:21:08.452172 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:21:08.452263 (+ 91us) client_negotiation.cc:769] Sending connection context
0618 22:21:08.452379 (+ 116us) client_negotiation.cc:240] Negotiation successful
0618 22:21:08.452521 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102}
I20260618 22:21:08.452770 16677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:08.443967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45686 (local address 127.0.98.66:39395)
0618 22:21:08.444346 (+ 379us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:08.444355 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:08.444371 (+ 16us) server_negotiation.cc:407] Connection header received
0618 22:21:08.444441 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:08.444449 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:08.444657 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:08.444843 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:21:08.445212 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:08.446067 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:08.446814 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:08.447281 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:08.447779 (+ 498us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:08.449463 (+ 1684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:21:08.449478 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:21:08.449488 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:21:08.449509 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:21:08.451062 (+ 1553us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:21:08.451474 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:21:08.451487 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:21:08.451489 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:21:08.451561 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:21:08.451832 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:21:08.451837 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:21:08.451839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:21:08.452058 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:21:08.452189 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:08.452456 (+ 267us) server_negotiation.cc:299] Negotiation successful
0618 22:21:08.452611 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":299,"thread_start_us":116,"threads_started":1}
I20260618 22:21:08.453609 16605 tablet_service.cc:1514] Processing DeleteTablet for tablet 147611d57207423683585fbc665e8bb5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:20:47 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:45686
I20260618 22:21:08.454586 16678 tablet_replica.cc:330] stopping tablet replica
I20260618 22:21:08.454921 16678 raft_consensus.cc:2229] T 147611d57207423683585fbc665e8bb5 P 610a350e9c644e61bd41f8646ed9f7c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:21:08.455188 16678 raft_consensus.cc:2258] T 147611d57207423683585fbc665e8bb5 P 610a350e9c644e61bd41f8646ed9f7c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:21:08.456342 16678 ts_tablet_manager.cc:1905] T 147611d57207423683585fbc665e8bb5 P 610a350e9c644e61bd41f8646ed9f7c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:21:08.460878 16678 ts_tablet_manager.cc:1918] T 147611d57207423683585fbc665e8bb5 P 610a350e9c644e61bd41f8646ed9f7c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:21:08.460973 16678 log.cc:1196] T 147611d57207423683585fbc665e8bb5 P 610a350e9c644e61bd41f8646ed9f7c7: Deleting WAL directory at /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-1/wal/wals/147611d57207423683585fbc665e8bb5
I20260618 22:21:08.461237 16678 ts_tablet_manager.cc:1939] T 147611d57207423683585fbc665e8bb5 P 610a350e9c644e61bd41f8646ed9f7c7: Deleting consensus metadata
I20260618 22:21:08.463124 16330 catalog_manager.cc:4841] TS 610a350e9c644e61bd41f8646ed9f7c7 (127.0.98.66:39395): tablet 147611d57207423683585fbc665e8bb5 (table test-table [id=b9a6fe0601c34747835651635cf8287a]) successfully deleted
W20260618 22:21:08.547576 16676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:21:08.547811 16676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:21:08.547851 16676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:21:08.552413 16676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:21:08.552467 16676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:21:08.552577 16676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260618 22:21:08.558461 16676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:38325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:44555
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=39675
--tserver_master_addrs=127.0.98.126:39001
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:21:08.559712 16676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:21:08.560702 16676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:21:08.570089 16683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:21:08.570163 16686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:21:08.570106 16684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:21:08.570600 16676 server_base.cc:1034] running on GCE node
I20260618 22:21:08.570961 16676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:21:08.571494 16676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:21:08.572682 16676 hybrid_clock.cc:648] HybridClock initialized: now 1781821268572650 us; error 46 us; skew 500 ppm
Jun 18 22:21:08 dist-test-slave-g6r4 krb5kdc[15764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821268, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:21:08.575325 16676 init.cc:377] Logged in from keytab as kudu/127.0.98.67@KRBTEST.COM (short username kudu)
I20260618 22:21:08.576280 16676 webserver.cc:460] Webserver started at http://127.0.98.67:39675/ using document root <none> and password file <none>
I20260618 22:21:08.576920 16676 fs_manager.cc:359] Metadata directory not provided
I20260618 22:21:08.577025 16676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:21:08.580817 16676 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:21:08.582860 16693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:08.584031 16676 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:21:08.584153 16676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-2/wal
uuid: "139063e1765148b181db5d13fdf5ff54"
format_stamp: "Formatted at 2026-06-18 22:20:47 on dist-test-slave-g6r4"
server_key: "0482da59d59c1f4c97da2193113e0cc9"
server_key_iv: "2fe7273eae39923eda6ccc37090ead2b"
server_key_version: "encryptionkey@0"
I20260618 22:21:08.584653 16676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:21:08.596895 16676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:21:08.599817 16676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:21:08.600013 16676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:21:08.600562 16676 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:21:08.601334 16676 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:21:08.601402 16676 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:08.601471 16676 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:21:08.601523 16676 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:08.610469 16676 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:44555
I20260618 22:21:08.610502 16806 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:44555 every 8 connection(s)
I20260618 22:21:08.611409 16676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
I20260618 22:21:08.619953 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 16676
Jun 18 22:21:08 dist-test-slave-g6r4 krb5kdc[15764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821268, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:21:08.625133 16534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:08.613214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:47961 (local address 127.0.98.126:39001)
0618 22:21:08.613361 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:08.613366 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:08.614302 (+ 936us) server_negotiation.cc:407] Connection header received
0618 22:21:08.615553 (+ 1251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:08.615557 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:08.615602 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:08.615677 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:21:08.616828 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:08.617360 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:08.618528 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:08.618749 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:08.618832 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:08.621608 (+ 2776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:21:08.621611 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:21:08.621613 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:21:08.621636 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:21:08.623143 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:21:08.623660 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:21:08.623664 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:21:08.623666 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:21:08.623745 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:21:08.624081 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:21:08.624084 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:21:08.624086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:21:08.624265 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:21:08.624371 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:08.624840 (+ 469us) server_negotiation.cc:299] Negotiation successful
0618 22:21:08.624955 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260618 22:21:08.626171 16809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:08.613661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39001 (local address 127.0.98.67:47961)
0618 22:21:08.614159 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:08.614202 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0618 22:21:08.615317 (+ 1115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:21:08.615839 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:21:08.615845 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:21:08.616211 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:21:08.616669 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:08.616682 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:08.617469 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:08.617471 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:08.618413 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:08.618422 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:08.618872 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:08.618875 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:08.619009 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:08.619612 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:21:08.619628 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:21:08.621395 (+ 1767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:21:08.623277 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:21:08.623283 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:21:08.623297 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:21:08.623558 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:21:08.623836 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:21:08.623840 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:21:08.623843 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:21:08.623979 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:21:08.624369 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:21:08.624376 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:21:08.624617 (+ 241us) client_negotiation.cc:769] Sending connection context
0618 22:21:08.624799 (+ 182us) client_negotiation.cc:240] Negotiation successful
0618 22:21:08.625106 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":112,"threads_started":1}
I20260618 22:21:08.627380 16807 heartbeater.cc:347] Connected to a master server at 127.0.98.126:39001
I20260618 22:21:08.627635 16807 heartbeater.cc:464] Registering TS with master...
I20260618 22:21:08.628146 16807 heartbeater.cc:511] Master 127.0.98.126:39001 requested a full tablet report, sending...
I20260618 22:21:08.628799 16810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:08.620672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56544 (local address 127.0.98.126:39001)
0618 22:21:08.620872 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:08.620875 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:08.621180 (+ 305us) server_negotiation.cc:407] Connection header received
0618 22:21:08.621342 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:08.621344 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:08.621394 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:08.621484 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:21:08.622270 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:08.622803 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:08.623694 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:08.623936 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:08.624046 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:08.625413 (+ 1367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:21:08.625416 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:21:08.625419 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:21:08.625442 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:21:08.627148 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:21:08.627609 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:21:08.627612 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:21:08.627613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:21:08.627699 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:21:08.627979 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:21:08.627982 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:21:08.627984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:21:08.628158 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:21:08.628354 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:08.628541 (+ 187us) server_negotiation.cc:299] Negotiation successful
0618 22:21:08.628629 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":76,"threads_started":1}
I20260618 22:21:08.629366 16344 ts_manager.cc:194] Registered new tserver with Master: 139063e1765148b181db5d13fdf5ff54 (127.0.98.67:44555)
I20260618 22:21:08.629997 16344 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.67@KRBTEST.COM'} at 127.0.98.67:47961
I20260618 22:21:08.630522 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 22:21:08.634223 16810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:08.631131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56546 (local address 127.0.98.126:39001)
0618 22:21:08.631235 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:08.631239 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:08.631461 (+ 222us) server_negotiation.cc:407] Connection header received
0618 22:21:08.631593 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:08.631595 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:08.631689 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:08.631768 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:21:08.632018 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:08.632473 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:08.633259 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:08.633461 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:08.633532 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:08.633769 (+ 237us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:21:08.633876 (+ 107us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0618 22:21:08.634119 (+ 243us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56546: token expired
Metrics: {"server-negotiator.queue_time_us":40}
W20260618 22:21:08.634282 16810 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56546: token expired
W20260618 22:21:08.634599 16273 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.126:39001)
I20260618 22:21:08.641259 16810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:08.634943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56562 (local address 127.0.98.126:39001)
0618 22:21:08.635061 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:08.635065 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:08.635078 (+ 13us) server_negotiation.cc:407] Connection header received
0618 22:21:08.635161 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:08.635164 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:08.635207 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:08.635268 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:21:08.635784 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:08.636220 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:08.636951 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:08.637140 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:08.637209 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:08.638395 (+ 1186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:21:08.638397 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:21:08.638399 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:21:08.638420 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:21:08.639821 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:21:08.640193 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:21:08.640195 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:21:08.640196 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:21:08.640265 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:21:08.640479 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:21:08.640480 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:21:08.640482 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:21:08.640639 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:21:08.640742 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:08.640972 (+ 230us) server_negotiation.cc:299] Negotiation successful
0618 22:21:08.641135 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260618 22:21:08.643848 16343 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56562:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 22:21:08.646960 16343 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:21:08.654060 16466 tablet_service.cc:1467] Processing CreateTablet for tablet 3b7ae90f164d4812924fa1d35f823d2b (DEFAULT_TABLE table=test-table [id=44661853848941a28dc791686f5125da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:21:08.654062 16605 tablet_service.cc:1467] Processing CreateTablet for tablet 3b7ae90f164d4812924fa1d35f823d2b (DEFAULT_TABLE table=test-table [id=44661853848941a28dc791686f5125da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:21:08.654603 16466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b7ae90f164d4812924fa1d35f823d2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:21:08.654632 16605 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b7ae90f164d4812924fa1d35f823d2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:21:08.657130 16424 tablet_bootstrap.cc:492] T 3b7ae90f164d4812924fa1d35f823d2b P ae5c1a50e67f4aed8830a55970810e8e: Bootstrap starting.
I20260618 22:21:08.657466 16563 tablet_bootstrap.cc:492] T 3b7ae90f164d4812924fa1d35f823d2b P 610a350e9c644e61bd41f8646ed9f7c7: Bootstrap starting.
I20260618 22:21:08.657994 16424 tablet_bootstrap.cc:654] T 3b7ae90f164d4812924fa1d35f823d2b P ae5c1a50e67f4aed8830a55970810e8e: Neither blocks nor log segments found. Creating new log.
I20260618 22:21:08.658629 16563 tablet_bootstrap.cc:654] T 3b7ae90f164d4812924fa1d35f823d2b P 610a350e9c644e61bd41f8646ed9f7c7: Neither blocks nor log segments found. Creating new log.
I20260618 22:21:08.658962 16539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:08.653130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:44555 (local address 127.0.0.1:45070)
0618 22:21:08.653440 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:08.653456 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 22:21:08.653527 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:21:08.654348 (+ 821us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:21:08.654351 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:21:08.654372 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:21:08.654472 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:08.654477 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:08.656020 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:08.656023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:08.657219 (+ 1196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:08.657230 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:08.658462 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:08.658465 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:08.658585 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:08.658643 (+ 58us) client_negotiation.cc:769] Sending connection context
0618 22:21:08.658727 (+ 84us) client_negotiation.cc:240] Negotiation successful
0618 22:21:08.658817 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234}
I20260618 22:21:08.659649 16816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:08.653433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45070 (local address 127.0.98.67:44555)
0618 22:21:08.653818 (+ 385us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:08.653830 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:08.653853 (+ 23us) server_negotiation.cc:407] Connection header received
0618 22:21:08.653951 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:08.653957 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:08.654174 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:08.654634 (+ 460us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:21:08.654806 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:08.655884 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:08.657406 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:08.658332 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:08.658464 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:08.658620 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:08.658971 (+ 351us) server_negotiation.cc:299] Negotiation successful
0618 22:21:08.659343 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":147,"threads_started":1}
I20260618 22:21:08.659780 16424 tablet_bootstrap.cc:492] T 3b7ae90f164d4812924fa1d35f823d2b P ae5c1a50e67f4aed8830a55970810e8e: No bootstrap required, opened a new log
I20260618 22:21:08.659866 16424 ts_tablet_manager.cc:1397] T 3b7ae90f164d4812924fa1d35f823d2b P ae5c1a50e67f4aed8830a55970810e8e: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260618 22:21:08.660228 16424 raft_consensus.cc:348] T 3b7ae90f164d4812924fa1d35f823d2b P ae5c1a50e67f4aed8830a55970810e8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139063e1765148b181db5d13fdf5ff54" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 44555 } } peers { permanent_uuid: "610a350e9c644e61bd41f8646ed9f7c7" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39395 } } peers { permanent_uuid: "ae5c1a50e67f4aed8830a55970810e8e" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33093 } }
I20260618 22:21:08.660318 16424 raft_consensus.cc:374] T 3b7ae90f164d4812924fa1d35f823d2b P ae5c1a50e67f4aed8830a55970810e8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:21:08.660344 16424 raft_consensus.cc:729] T 3b7ae90f164d4812924fa1d35f823d2b P ae5c1a50e67f4aed8830a55970810e8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae5c1a50e67f4aed8830a55970810e8e, State: Initialized, Role: FOLLOWER
I20260618 22:21:08.660494 16424 consensus_queue.cc:260] T 3b7ae90f164d4812924fa1d35f823d2b P ae5c1a50e67f4aed8830a55970810e8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139063e1765148b181db5d13fdf5ff54" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 44555 } } peers { permanent_uuid: "610a350e9c644e61bd41f8646ed9f7c7" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39395 } } peers { permanent_uuid: "ae5c1a50e67f4aed8830a55970810e8e" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33093 } }
I20260618 22:21:08.660745 16424 ts_tablet_manager.cc:1428] T 3b7ae90f164d4812924fa1d35f823d2b P ae5c1a50e67f4aed8830a55970810e8e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260618 22:21:08.660838 16563 tablet_bootstrap.cc:492] T 3b7ae90f164d4812924fa1d35f823d2b P 610a350e9c644e61bd41f8646ed9f7c7: No bootstrap required, opened a new log
I20260618 22:21:08.660907 16563 ts_tablet_manager.cc:1397] T 3b7ae90f164d4812924fa1d35f823d2b P 610a350e9c644e61bd41f8646ed9f7c7: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260618 22:21:08.661303 16563 raft_consensus.cc:348] T 3b7ae90f164d4812924fa1d35f823d2b P 610a350e9c644e61bd41f8646ed9f7c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139063e1765148b181db5d13fdf5ff54" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 44555 } } peers { permanent_uuid: "610a350e9c644e61bd41f8646ed9f7c7" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39395 } } peers { permanent_uuid: "ae5c1a50e67f4aed8830a55970810e8e" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33093 } }
I20260618 22:21:08.661399 16563 raft_consensus.cc:374] T 3b7ae90f164d4812924fa1d35f823d2b P 610a350e9c644e61bd41f8646ed9f7c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:21:08.661433 16563 raft_consensus.cc:729] T 3b7ae90f164d4812924fa1d35f823d2b P 610a350e9c644e61bd41f8646ed9f7c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 610a350e9c644e61bd41f8646ed9f7c7, State: Initialized, Role: FOLLOWER
I20260618 22:21:08.661548 16563 consensus_queue.cc:260] T 3b7ae90f164d4812924fa1d35f823d2b P 610a350e9c644e61bd41f8646ed9f7c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139063e1765148b181db5d13fdf5ff54" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 44555 } } peers { permanent_uuid: "610a350e9c644e61bd41f8646ed9f7c7" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39395 } } peers { permanent_uuid: "ae5c1a50e67f4aed8830a55970810e8e" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33093 } }
I20260618 22:21:08.661773 16563 ts_tablet_manager.cc:1428] T 3b7ae90f164d4812924fa1d35f823d2b P 610a350e9c644e61bd41f8646ed9f7c7: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260618 22:21:08.662693 16741 tablet_service.cc:1467] Processing CreateTablet for tablet 3b7ae90f164d4812924fa1d35f823d2b (DEFAULT_TABLE table=test-table [id=44661853848941a28dc791686f5125da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:21:08.663766 16741 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b7ae90f164d4812924fa1d35f823d2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:21:08.670243 16819 tablet_bootstrap.cc:492] T 3b7ae90f164d4812924fa1d35f823d2b P 139063e1765148b181db5d13fdf5ff54: Bootstrap starting.
I20260618 22:21:08.672560 16819 tablet_bootstrap.cc:654] T 3b7ae90f164d4812924fa1d35f823d2b P 139063e1765148b181db5d13fdf5ff54: Neither blocks nor log segments found. Creating new log.
I20260618 22:21:08.673223 16819 log.cc:824] T 3b7ae90f164d4812924fa1d35f823d2b P 139063e1765148b181db5d13fdf5ff54: Log is configured to *not* fsync() on all Append() calls
W20260618 22:21:08.674551 16673 tablet.cc:2369] T 3b7ae90f164d4812924fa1d35f823d2b P 610a350e9c644e61bd41f8646ed9f7c7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:21:08.675091 16819 tablet_bootstrap.cc:492] T 3b7ae90f164d4812924fa1d35f823d2b P 139063e1765148b181db5d13fdf5ff54: No bootstrap required, opened a new log
I20260618 22:21:08.675244 16819 ts_tablet_manager.cc:1397] T 3b7ae90f164d4812924fa1d35f823d2b P 139063e1765148b181db5d13fdf5ff54: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260618 22:21:08.678699 16819 raft_consensus.cc:348] T 3b7ae90f164d4812924fa1d35f823d2b P 139063e1765148b181db5d13fdf5ff54 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139063e1765148b181db5d13fdf5ff54" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 44555 } } peers { permanent_uuid: "610a350e9c644e61bd41f8646ed9f7c7" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39395 } } peers { permanent_uuid: "ae5c1a50e67f4aed8830a55970810e8e" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33093 } }
I20260618 22:21:08.678934 16819 raft_consensus.cc:374] T 3b7ae90f164d4812924fa1d35f823d2b P 139063e1765148b181db5d13fdf5ff54 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:21:08.678968 16819 raft_consensus.cc:729] T 3b7ae90f164d4812924fa1d35f823d2b P 139063e1765148b181db5d13fdf5ff54 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 139063e1765148b181db5d13fdf5ff54, State: Initialized, Role: FOLLOWER
I20260618 22:21:08.679363 16819 consensus_queue.cc:260] T 3b7ae90f164d4812924fa1d35f823d2b P 139063e1765148b181db5d13fdf5ff54 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139063e1765148b181db5d13fdf5ff54" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 44555 } } peers { permanent_uuid: "610a350e9c644e61bd41f8646ed9f7c7" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39395 } } peers { permanent_uuid: "ae5c1a50e67f4aed8830a55970810e8e" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33093 } }
I20260618 22:21:08.679946 16807 heartbeater.cc:503] Master 127.0.98.126:39001 was elected leader, sending a full tablet report...
I20260618 22:21:08.680251 16819 ts_tablet_manager.cc:1428] T 3b7ae90f164d4812924fa1d35f823d2b P 139063e1765148b181db5d13fdf5ff54: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
W20260618 22:21:08.726470 16533 tablet.cc:2369] T 3b7ae90f164d4812924fa1d35f823d2b P ae5c1a50e67f4aed8830a55970810e8e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:21:08.793773 16538 raft_consensus.cc:482] T 3b7ae90f164d4812924fa1d35f823d2b P ae5c1a50e67f4aed8830a55970810e8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:21:08.794044 16538 raft_consensus.cc:504] T 3b7ae90f164d4812924fa1d35f823d2b P ae5c1a50e67f4aed8830a55970810e8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139063e1765148b181db5d13fdf5ff54" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 44555 } } peers { permanent_uuid: "610a350e9c644e61bd41f8646ed9f7c7" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39395 } } peers { permanent_uuid: "ae5c1a50e67f4aed8830a55970810e8e" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33093 } }
I20260618 22:21:08.795361 16538 leader_election.cc:288] T 3b7ae90f164d4812924fa1d35f823d2b P ae5c1a50e67f4aed8830a55970810e8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 139063e1765148b181db5d13fdf5ff54 (127.0.98.67:44555), 610a350e9c644e61bd41f8646ed9f7c7 (127.0.98.66:39395)
I20260618 22:21:08.800551 16816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:08.795679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:35777 (local address 127.0.98.67:44555)
0618 22:21:08.795979 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:08.795984 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:08.796070 (+ 86us) server_negotiation.cc:407] Connection header received
0618 22:21:08.796116 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:08.796119 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:08.796165 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:08.796243 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:21:08.796648 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:08.797451 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:08.799266 (+ 1815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:08.799854 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:08.799978 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:08.800084 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:08.800348 (+ 264us) server_negotiation.cc:299] Negotiation successful
0618 22:21:08.800419 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":230}
I20260618 22:21:08.800568 16677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:08.795844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:49521 (local address 127.0.98.66:39395)
0618 22:21:08.795978 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:08.795983 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:08.796001 (+ 18us) server_negotiation.cc:407] Connection header received
0618 22:21:08.796092 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:08.796096 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:08.796159 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:08.796257 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:21:08.796648 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:08.797450 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:08.799266 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:08.799922 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:08.800014 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:08.800172 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:08.800348 (+ 176us) server_negotiation.cc:299] Negotiation successful
0618 22:21:08.800419 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260618 22:21:08.800616 16823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:08.795674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:39395 (local address 127.0.98.65:49521)
0618 22:21:08.795920 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:08.795931 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 22:21:08.795993 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:21:08.796285 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:21:08.796288 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:21:08.796331 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:21:08.796525 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:08.796532 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:08.797570 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:08.797574 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:08.799128 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:08.799139 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:08.800020 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:08.800023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:08.800165 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:08.800233 (+ 68us) client_negotiation.cc:769] Sending connection context
0618 22:21:08.800315 (+ 82us) client_negotiation.cc:240] Negotiation successful
0618 22:21:08.800418 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":240,"spinlock_wait_cycles":6656,"thread_start_us":65,"threads_started":1}
I20260618 22:21:08.800625 16822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:08.795575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:44555 (local address 127.0.98.65:35777)
0618 22:21:08.795836 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:08.795851 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:21:08.795969 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:21:08.796285 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:21:08.796288 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:21:08.796329 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:21:08.796525 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:08.796532 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:08.797570 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:08.797574 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:08.799128 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:08.799139 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:08.799995 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:08.800000 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:08.800156 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:08.800241 (+ 85us) client_negotiation.cc:769] Sending connection context
0618 22:21:08.800315 (+ 74us) client_negotiation.cc:240] Negotiation successful
0618 22:21:08.800418 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":265,"spinlock_wait_cycles":38272,"thread_start_us":90,"threads_started":1}
I20260618 22:21:08.801265 16625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b7ae90f164d4812924fa1d35f823d2b" candidate_uuid: "ae5c1a50e67f4aed8830a55970810e8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "610a350e9c644e61bd41f8646ed9f7c7" is_pre_election: true
I20260618 22:21:08.801609 16625 raft_consensus.cc:2454] T 3b7ae90f164d4812924fa1d35f823d2b P 610a350e9c644e61bd41f8646ed9f7c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae5c1a50e67f4aed8830a55970810e8e in term 0.
I20260618 22:21:08.801577 16761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b7ae90f164d4812924fa1d35f823d2b" candidate_uuid: "ae5c1a50e67f4aed8830a55970810e8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "139063e1765148b181db5d13fdf5ff54" is_pre_election: true
I20260618 22:21:08.801884 16761 raft_consensus.cc:2454] T 3b7ae90f164d4812924fa1d35f823d2b P 139063e1765148b181db5d13fdf5ff54 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae5c1a50e67f4aed8830a55970810e8e in term 0.
I20260618 22:21:08.802091 16420 leader_election.cc:302] T 3b7ae90f164d4812924fa1d35f823d2b P ae5c1a50e67f4aed8830a55970810e8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 610a350e9c644e61bd41f8646ed9f7c7, ae5c1a50e67f4aed8830a55970810e8e; no voters:
I20260618 22:21:08.802418 16538 raft_consensus.cc:2781] T 3b7ae90f164d4812924fa1d35f823d2b P ae5c1a50e67f4aed8830a55970810e8e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:21:08.802496 16538 raft_consensus.cc:482] T 3b7ae90f164d4812924fa1d35f823d2b P ae5c1a50e67f4aed8830a55970810e8e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:21:08.802531 16538 raft_consensus.cc:3037] T 3b7ae90f164d4812924fa1d35f823d2b P ae5c1a50e67f4aed8830a55970810e8e [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:21:08.803699 16538 raft_consensus.cc:504] T 3b7ae90f164d4812924fa1d35f823d2b P ae5c1a50e67f4aed8830a55970810e8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139063e1765148b181db5d13fdf5ff54" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 44555 } } peers { permanent_uuid: "610a350e9c644e61bd41f8646ed9f7c7" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39395 } } peers { permanent_uuid: "ae5c1a50e67f4aed8830a55970810e8e" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33093 } }
I20260618 22:21:08.804088 16538 leader_election.cc:288] T 3b7ae90f164d4812924fa1d35f823d2b P ae5c1a50e67f4aed8830a55970810e8e [CANDIDATE]: Term 1 election: Requested vote from peers 139063e1765148b181db5d13fdf5ff54 (127.0.98.67:44555), 610a350e9c644e61bd41f8646ed9f7c7 (127.0.98.66:39395)
I20260618 22:21:08.804395 16761 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b7ae90f164d4812924fa1d35f823d2b" candidate_uuid: "ae5c1a50e67f4aed8830a55970810e8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "139063e1765148b181db5d13fdf5ff54"
I20260618 22:21:08.804495 16625 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b7ae90f164d4812924fa1d35f823d2b" candidate_uuid: "ae5c1a50e67f4aed8830a55970810e8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "610a350e9c644e61bd41f8646ed9f7c7"
I20260618 22:21:08.804514 16761 raft_consensus.cc:3037] T 3b7ae90f164d4812924fa1d35f823d2b P 139063e1765148b181db5d13fdf5ff54 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:21:08.804651 16625 raft_consensus.cc:3037] T 3b7ae90f164d4812924fa1d35f823d2b P 610a350e9c644e61bd41f8646ed9f7c7 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:21:08.805568 16761 raft_consensus.cc:2454] T 3b7ae90f164d4812924fa1d35f823d2b P 139063e1765148b181db5d13fdf5ff54 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae5c1a50e67f4aed8830a55970810e8e in term 1.
I20260618 22:21:08.805696 16625 raft_consensus.cc:2454] T 3b7ae90f164d4812924fa1d35f823d2b P 610a350e9c644e61bd41f8646ed9f7c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae5c1a50e67f4aed8830a55970810e8e in term 1.
I20260618 22:21:08.805876 16418 leader_election.cc:302] T 3b7ae90f164d4812924fa1d35f823d2b P ae5c1a50e67f4aed8830a55970810e8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 139063e1765148b181db5d13fdf5ff54, ae5c1a50e67f4aed8830a55970810e8e; no voters:
I20260618 22:21:08.806078 16538 raft_consensus.cc:2781] T 3b7ae90f164d4812924fa1d35f823d2b P ae5c1a50e67f4aed8830a55970810e8e [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:21:08.806243 16538 raft_consensus.cc:686] T 3b7ae90f164d4812924fa1d35f823d2b P ae5c1a50e67f4aed8830a55970810e8e [term 1 LEADER]: Becoming Leader. State: Replica: ae5c1a50e67f4aed8830a55970810e8e, State: Running, Role: LEADER
I20260618 22:21:08.806506 16538 consensus_queue.cc:237] T 3b7ae90f164d4812924fa1d35f823d2b P ae5c1a50e67f4aed8830a55970810e8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139063e1765148b181db5d13fdf5ff54" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 44555 } } peers { permanent_uuid: "610a350e9c644e61bd41f8646ed9f7c7" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39395 } } peers { permanent_uuid: "ae5c1a50e67f4aed8830a55970810e8e" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33093 } }
I20260618 22:21:08.809428 16344 catalog_manager.cc:5495] T 3b7ae90f164d4812924fa1d35f823d2b P ae5c1a50e67f4aed8830a55970810e8e reported cstate change: term changed from 0 to 1, leader changed from <none> to ae5c1a50e67f4aed8830a55970810e8e (127.0.98.65). New cstate: current_term: 1 leader_uuid: "ae5c1a50e67f4aed8830a55970810e8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139063e1765148b181db5d13fdf5ff54" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 44555 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "610a350e9c644e61bd41f8646ed9f7c7" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ae5c1a50e67f4aed8830a55970810e8e" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33093 } health_report { overall_health: HEALTHY } } }
I20260618 22:21:08.848536 16827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:08.844882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39172 (local address 127.0.98.65:33093)
0618 22:21:08.845193 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:08.845200 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:08.845211 (+ 11us) server_negotiation.cc:407] Connection header received
0618 22:21:08.845278 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:08.845280 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:08.845330 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:08.845460 (+ 130us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:21:08.845726 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:08.846255 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:08.847087 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:08.847323 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:08.847435 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:08.847711 (+ 276us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:21:08.848119 (+ 408us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:21:08.848251 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:08.848313 (+ 62us) server_negotiation.cc:299] Negotiation successful
0618 22:21:08.848366 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242,"thread_start_us":99,"threads_started":1}
I20260618 22:21:08.852375 16625 raft_consensus.cc:1261] T 3b7ae90f164d4812924fa1d35f823d2b P 610a350e9c644e61bd41f8646ed9f7c7 [term 1 FOLLOWER]: Refusing update from remote peer ae5c1a50e67f4aed8830a55970810e8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:21:08.852385 16761 raft_consensus.cc:1261] T 3b7ae90f164d4812924fa1d35f823d2b P 139063e1765148b181db5d13fdf5ff54 [term 1 FOLLOWER]: Refusing update from remote peer ae5c1a50e67f4aed8830a55970810e8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:21:08.853086 16824 consensus_queue.cc:1035] T 3b7ae90f164d4812924fa1d35f823d2b P ae5c1a50e67f4aed8830a55970810e8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "610a350e9c644e61bd41f8646ed9f7c7" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:21:08.853303 16538 consensus_queue.cc:1035] T 3b7ae90f164d4812924fa1d35f823d2b P ae5c1a50e67f4aed8830a55970810e8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "139063e1765148b181db5d13fdf5ff54" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 44555 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:21:08.858052 16828 mvcc.cc:205] Tried to move back new op lower bound from 7298339917212913664 to 7298339917034053632. Current Snapshot: MvccSnapshot[applied={T|T < 7298339917212913664}]
I20260618 22:21:08.866632 16343 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:21:08.866762 16343 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:21:08.868961 16343 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 56d1566a923f47dca11d653707ba497a: Sending DeleteTablet for 3 replicas of tablet 3b7ae90f164d4812924fa1d35f823d2b
I20260618 22:21:08.869566 16605 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b7ae90f164d4812924fa1d35f823d2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:21:08 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:45686
I20260618 22:21:08.869621 16466 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b7ae90f164d4812924fa1d35f823d2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:21:08 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:39160
I20260618 22:21:08.869616 16741 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b7ae90f164d4812924fa1d35f823d2b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:21:08 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:45070
I20260618 22:21:08.869807 16678 tablet_replica.cc:330] T 3b7ae90f164d4812924fa1d35f823d2b P 610a350e9c644e61bd41f8646ed9f7c7: stopping tablet replica
I20260618 22:21:08.869941 16678 raft_consensus.cc:2229] T 3b7ae90f164d4812924fa1d35f823d2b P 610a350e9c644e61bd41f8646ed9f7c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:21:08.870045 16839 tablet_replica.cc:330] T 3b7ae90f164d4812924fa1d35f823d2b P 139063e1765148b181db5d13fdf5ff54: stopping tablet replica
I20260618 22:21:08.870071 16678 raft_consensus.cc:2258] T 3b7ae90f164d4812924fa1d35f823d2b P 610a350e9c644e61bd41f8646ed9f7c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:21:08.870234 16838 tablet_replica.cc:330] T 3b7ae90f164d4812924fa1d35f823d2b P ae5c1a50e67f4aed8830a55970810e8e: stopping tablet replica
I20260618 22:21:08.870332 16838 raft_consensus.cc:2229] T 3b7ae90f164d4812924fa1d35f823d2b P ae5c1a50e67f4aed8830a55970810e8e [term 1 LEADER]: Raft consensus shutting down.
I20260618 22:21:08.870322 16839 raft_consensus.cc:2229] T 3b7ae90f164d4812924fa1d35f823d2b P 139063e1765148b181db5d13fdf5ff54 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:21:08.870559 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 16382
I20260618 22:21:08.870587 16838 raft_consensus.cc:2258] T 3b7ae90f164d4812924fa1d35f823d2b P ae5c1a50e67f4aed8830a55970810e8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:21:08.870635 16839 raft_consensus.cc:2258] T 3b7ae90f164d4812924fa1d35f823d2b P 139063e1765148b181db5d13fdf5ff54 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:21:08.873081 16678 ts_tablet_manager.cc:1905] T 3b7ae90f164d4812924fa1d35f823d2b P 610a350e9c644e61bd41f8646ed9f7c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:21:08.873176 16839 ts_tablet_manager.cc:1905] T 3b7ae90f164d4812924fa1d35f823d2b P 139063e1765148b181db5d13fdf5ff54: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:21:08.875324 16839 ts_tablet_manager.cc:1918] T 3b7ae90f164d4812924fa1d35f823d2b P 139063e1765148b181db5d13fdf5ff54: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:21:08.875445 16839 log.cc:1196] T 3b7ae90f164d4812924fa1d35f823d2b P 139063e1765148b181db5d13fdf5ff54: Deleting WAL directory at /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-2/wal/wals/3b7ae90f164d4812924fa1d35f823d2b
I20260618 22:21:08.875782 16839 ts_tablet_manager.cc:1939] T 3b7ae90f164d4812924fa1d35f823d2b P 139063e1765148b181db5d13fdf5ff54: Deleting consensus metadata
I20260618 22:21:08.876195 16678 ts_tablet_manager.cc:1918] T 3b7ae90f164d4812924fa1d35f823d2b P 610a350e9c644e61bd41f8646ed9f7c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:21:08.876273 16678 log.cc:1196] T 3b7ae90f164d4812924fa1d35f823d2b P 610a350e9c644e61bd41f8646ed9f7c7: Deleting WAL directory at /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-1/wal/wals/3b7ae90f164d4812924fa1d35f823d2b
I20260618 22:21:08.876549 16678 ts_tablet_manager.cc:1939] T 3b7ae90f164d4812924fa1d35f823d2b P 610a350e9c644e61bd41f8646ed9f7c7: Deleting consensus metadata
I20260618 22:21:08.877105 16328 catalog_manager.cc:4841] TS 139063e1765148b181db5d13fdf5ff54 (127.0.98.67:44555): tablet 3b7ae90f164d4812924fa1d35f823d2b (table test-table [id=44661853848941a28dc791686f5125da]) successfully deleted
I20260618 22:21:08.877876 16330 catalog_manager.cc:4841] TS 610a350e9c644e61bd41f8646ed9f7c7 (127.0.98.66:39395): tablet 3b7ae90f164d4812924fa1d35f823d2b (table test-table [id=44661853848941a28dc791686f5125da]) successfully deleted
W20260618 22:21:08.878304 16329 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.65:33093 (error 108)
I20260618 22:21:08.879171 16539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:08.878964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:33093 (local address 127.0.0.1:39176)
0618 22:21:08.879047 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:08.879107 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.65:33093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260618 22:21:08.879331 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 16537
W20260618 22:21:08.879406 16329 catalog_manager.cc:4584] TS ae5c1a50e67f4aed8830a55970810e8e (127.0.98.65:33093): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3b7ae90f164d4812924fa1d35f823d2b: Network error: Client connection negotiation failed: client connection to 127.0.98.65:33093: connect: Connection refused (error 111)
I20260618 22:21:08.884884 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 16676
I20260618 22:21:08.890445 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 16311
I20260618 22:21:28.897060 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:39001
--webserver_interface=127.0.98.126
--webserver_port=37291
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:38325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:39001
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:21:29.006559 16841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:21:29.006834 16841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:21:29.006913 16841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:21:29.010877 16841 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260618 22:21:29.010944 16841 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260618 22:21:29.010972 16841 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260618 22:21:29.011000 16841 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:21:29.011044 16841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:21:29.011080 16841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:21:29.011142 16841 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:21:29.011180 16841 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:21:29.016323 16841 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:38325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:39001
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:39001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=37291
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:21:29.017287 16841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:21:29.018296 16841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:21:29.026557 16847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:21:29.026575 16846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:21:29.026607 16849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:21:29.026786 16841 server_base.cc:1034] running on GCE node
I20260618 22:21:29.027405 16841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:21:29.028417 16841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:21:29.029601 16841 hybrid_clock.cc:648] HybridClock initialized: now 1781821289029586 us; error 34 us; skew 500 ppm
Jun 18 22:21:29 dist-test-slave-g6r4 krb5kdc[15764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821289, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:21:29.036547 16841 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260618 22:21:29.037441 16841 webserver.cc:460] Webserver started at http://127.0.98.126:37291/ using document root <none> and password file <none>
I20260618 22:21:29.038038 16841 fs_manager.cc:359] Metadata directory not provided
I20260618 22:21:29.038143 16841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:21:29.041745 16841 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.002s
I20260618 22:21:29.043850 16856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:29.045003 16841 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:21:29.045163 16841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/master-0/wal
uuid: "56d1566a923f47dca11d653707ba497a"
format_stamp: "Formatted at 2026-06-18 22:20:46 on dist-test-slave-g6r4"
server_key: "339cc53340c9db76e65087b522b03ba8"
server_key_iv: "9a85caacb642db088086e741d0d79546"
server_key_version: "encryptionkey@0"
I20260618 22:21:29.045672 16841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:21:29.062729 16841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:21:29.065989 16841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:21:29.066183 16841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:21:29.073244 16841 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:39001
I20260618 22:21:29.073516 16908 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:39001 every 8 connection(s)
I20260618 22:21:29.074326 16841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
I20260618 22:21:29.078414 16909 sys_catalog.cc:263] Verifying existing consensus state
I20260618 22:21:29.080006 16909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56d1566a923f47dca11d653707ba497a: Bootstrap starting.
I20260618 22:21:29.084152 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 16841
I20260618 22:21:29.084618 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:33093
--local_ip_for_outbound_sockets=127.0.98.65
--tserver_master_addrs=127.0.98.126:39001
--webserver_port=35901
--webserver_interface=127.0.98.65
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:38325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 22:21:29.097433 16909 log.cc:824] T 00000000000000000000000000000000 P 56d1566a923f47dca11d653707ba497a: Log is configured to *not* fsync() on all Append() calls
I20260618 22:21:29.108281 16909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56d1566a923f47dca11d653707ba497a: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260618 22:21:29.108676 16909 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56d1566a923f47dca11d653707ba497a: Bootstrap complete.
I20260618 22:21:29.112685 16909 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56d1566a923f47dca11d653707ba497a [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d1566a923f47dca11d653707ba497a" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39001 } }
I20260618 22:21:29.113083 16909 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56d1566a923f47dca11d653707ba497a [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56d1566a923f47dca11d653707ba497a, State: Initialized, Role: FOLLOWER
I20260618 22:21:29.113595 16909 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56d1566a923f47dca11d653707ba497a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d1566a923f47dca11d653707ba497a" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39001 } }
I20260618 22:21:29.113740 16909 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56d1566a923f47dca11d653707ba497a [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:21:29.113804 16909 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56d1566a923f47dca11d653707ba497a [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:21:29.113945 16909 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56d1566a923f47dca11d653707ba497a [term 2 FOLLOWER]: Advancing to term 3
I20260618 22:21:29.116107 16909 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56d1566a923f47dca11d653707ba497a [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d1566a923f47dca11d653707ba497a" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39001 } }
I20260618 22:21:29.116452 16909 leader_election.cc:302] T 00000000000000000000000000000000 P 56d1566a923f47dca11d653707ba497a [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56d1566a923f47dca11d653707ba497a; no voters:
I20260618 22:21:29.116755 16909 leader_election.cc:288] T 00000000000000000000000000000000 P 56d1566a923f47dca11d653707ba497a [CANDIDATE]: Term 3 election: Requested vote from peers
I20260618 22:21:29.117012 16913 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56d1566a923f47dca11d653707ba497a [term 3 FOLLOWER]: Leader election won for term 3
I20260618 22:21:29.117312 16913 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56d1566a923f47dca11d653707ba497a [term 3 LEADER]: Becoming Leader. State: Replica: 56d1566a923f47dca11d653707ba497a, State: Running, Role: LEADER
I20260618 22:21:29.117682 16913 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56d1566a923f47dca11d653707ba497a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d1566a923f47dca11d653707ba497a" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39001 } }
I20260618 22:21:29.117843 16909 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56d1566a923f47dca11d653707ba497a [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:21:29.119212 16915 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56d1566a923f47dca11d653707ba497a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56d1566a923f47dca11d653707ba497a. Latest consensus state: current_term: 3 leader_uuid: "56d1566a923f47dca11d653707ba497a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d1566a923f47dca11d653707ba497a" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39001 } } }
I20260618 22:21:29.119330 16915 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56d1566a923f47dca11d653707ba497a [sys.catalog]: This master's current role is: LEADER
I20260618 22:21:29.119665 16914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56d1566a923f47dca11d653707ba497a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "56d1566a923f47dca11d653707ba497a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56d1566a923f47dca11d653707ba497a" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39001 } } }
I20260618 22:21:29.119738 16923 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:21:29.119764 16914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56d1566a923f47dca11d653707ba497a [sys.catalog]: This master's current role is: LEADER
I20260618 22:21:29.125392 16923 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:21:29.126169 16923 catalog_manager.cc:1259] Loaded cluster ID: 1146874a6ad643a9a163dfb9a1560f69
I20260618 22:21:29.126293 16923 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:21:29.129058 16923 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:21:29.130285 16923 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 56d1566a923f47dca11d653707ba497a: Loaded TSK: 0
I20260618 22:21:29.130465 16923 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 56d1566a923f47dca11d653707ba497a: Loaded TSK: 1
F20260618 22:21:29.131289 16923 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f7d67c579ad google::LogMessage::Fail() at ??:0
@ 0x7f7d67c5a88a google::LogMessage::SendToLog() at ??:0
@ 0x7f7d67c574de google::LogMessage::Flush() at ??:0
@ 0x7f7d67c58a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f7d6a91b18d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f7d6a913f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f7d725bc7bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f7d7259699c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f7d72597178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f7d725d8df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f7d725f782b std::function<>::operator()() at ??:0
@ 0x7f7d72596b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f7d72597f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f7d72594579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f7d725d8640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f7d69ea93d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f7d69ea9c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f7d69eab3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f7d69e9a512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f7d6b4de6db start_thread at ??:0
@ 0x7f7d6675561f clone at ??:0
W20260618 22:21:29.229471 16911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:21:29.229702 16911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:21:29.229745 16911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:21:29.234370 16911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:21:29.234417 16911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:21:29.234478 16911 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260618 22:21:29.240001 16911 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:38325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:33093
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=35901
--tserver_master_addrs=127.0.98.126:39001
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16911
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:21:29.241093 16911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:21:29.242329 16911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:21:29.252177 16957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:21:29.252128 16956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:21:29.252447 16911 server_base.cc:1034] running on GCE node
W20260618 22:21:29.252141 16959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:21:29.252965 16911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:21:29.253568 16911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:21:29.254752 16911 hybrid_clock.cc:648] HybridClock initialized: now 1781821289254699 us; error 73 us; skew 500 ppm
Jun 18 22:21:29 dist-test-slave-g6r4 krb5kdc[15764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821289, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:21:29.257617 16911 init.cc:377] Logged in from keytab as kudu/127.0.98.65@KRBTEST.COM (short username kudu)
I20260618 22:21:29.258496 16911 webserver.cc:460] Webserver started at http://127.0.98.65:35901/ using document root <none> and password file <none>
I20260618 22:21:29.259056 16911 fs_manager.cc:359] Metadata directory not provided
I20260618 22:21:29.259148 16911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:21:29.263022 16911 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:21:29.265224 16966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:29.266613 16911 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260618 22:21:29.266746 16911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-0/wal
uuid: "ae5c1a50e67f4aed8830a55970810e8e"
format_stamp: "Formatted at 2026-06-18 22:20:47 on dist-test-slave-g6r4"
server_key: "0eea59f8f5fe730b371cff829746fe6f"
server_key_iv: "681f3e6a2126ea0e2d2a237f13dc42d0"
server_key_version: "encryptionkey@0"
I20260618 22:21:29.267274 16911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:21:29.282156 16911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:21:29.285425 16911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:21:29.285630 16911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:21:29.286286 16911 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:21:29.287405 16973 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260618 22:21:29.293102 16911 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260618 22:21:29.293176 16911 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260618 22:21:29.293277 16911 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260618 22:21:29.294909 16911 ts_tablet_manager.cc:610] Registered 1 tablets
I20260618 22:21:29.294975 16911 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260618 22:21:29.295287 16973 tablet_bootstrap.cc:492] T 3b7ae90f164d4812924fa1d35f823d2b P ae5c1a50e67f4aed8830a55970810e8e: Bootstrap starting.
I20260618 22:21:29.305554 16911 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:33093
I20260618 22:21:29.305783 17080 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:33093 every 8 connection(s)
I20260618 22:21:29.306557 16911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
I20260618 22:21:29.311298 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 16911
I20260618 22:21:29.311736 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:39395
--local_ip_for_outbound_sockets=127.0.98.66
--tserver_master_addrs=127.0.98.126:39001
--webserver_port=39183
--webserver_interface=127.0.98.66
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:38325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 22:21:29.319051 16973 log.cc:824] T 3b7ae90f164d4812924fa1d35f823d2b P ae5c1a50e67f4aed8830a55970810e8e: Log is configured to *not* fsync() on all Append() calls
I20260618 22:21:29.328445 16973 tablet_bootstrap.cc:492] T 3b7ae90f164d4812924fa1d35f823d2b P ae5c1a50e67f4aed8830a55970810e8e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260618 22:21:29.328902 16973 tablet_bootstrap.cc:492] T 3b7ae90f164d4812924fa1d35f823d2b P ae5c1a50e67f4aed8830a55970810e8e: Bootstrap complete.
I20260618 22:21:29.330520 16973 ts_tablet_manager.cc:1397] T 3b7ae90f164d4812924fa1d35f823d2b P ae5c1a50e67f4aed8830a55970810e8e: Time spent bootstrapping tablet: real 0.035s user 0.026s sys 0.008s
I20260618 22:21:29.333925 16973 raft_consensus.cc:348] T 3b7ae90f164d4812924fa1d35f823d2b P ae5c1a50e67f4aed8830a55970810e8e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139063e1765148b181db5d13fdf5ff54" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 44555 } } peers { permanent_uuid: "610a350e9c644e61bd41f8646ed9f7c7" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39395 } } peers { permanent_uuid: "ae5c1a50e67f4aed8830a55970810e8e" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33093 } }
I20260618 22:21:29.334503 16973 raft_consensus.cc:729] T 3b7ae90f164d4812924fa1d35f823d2b P ae5c1a50e67f4aed8830a55970810e8e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae5c1a50e67f4aed8830a55970810e8e, State: Initialized, Role: FOLLOWER
I20260618 22:21:29.335031 16973 consensus_queue.cc:260] T 3b7ae90f164d4812924fa1d35f823d2b P ae5c1a50e67f4aed8830a55970810e8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139063e1765148b181db5d13fdf5ff54" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 44555 } } peers { permanent_uuid: "610a350e9c644e61bd41f8646ed9f7c7" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39395 } } peers { permanent_uuid: "ae5c1a50e67f4aed8830a55970810e8e" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33093 } }
I20260618 22:21:29.336071 16973 ts_tablet_manager.cc:1428] T 3b7ae90f164d4812924fa1d35f823d2b P ae5c1a50e67f4aed8830a55970810e8e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
W20260618 22:21:29.393208 17083 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:21:29.308434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39001 (local address 127.0.98.65:36645)
0618 22:21:29.308894 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:29.308944 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0618 22:21:29.309732 (+ 788us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:21:29.392232 (+ 82500us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39001: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":137,"threads_started":1}
W20260618 22:21:29.393784 17081 heartbeater.cc:644] Failed to heartbeat to 127.0.98.126:39001 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.126:39001: Client connection negotiation failed: client connection to 127.0.98.126:39001: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260618 22:21:29.394593 17083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:29.394301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39001 (local address 127.0.98.65:40825)
0618 22:21:29.394444 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:29.394523 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260618 22:21:29.395282 17083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:29.395059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39001 (local address 127.0.98.65:33905)
0618 22:21:29.395173 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:29.395225 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260618 22:21:29.395475 17081 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260618 22:21:29.420745 17084 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:21:29.420989 17084 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:21:29.421062 17084 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:21:29.425173 17084 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:21:29.425227 17084 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:21:29.425325 17084 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260618 22:21:29.430491 17084 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:38325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:39395
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=39183
--tserver_master_addrs=127.0.98.126:39001
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17084
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:21:29.431541 17084 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:21:29.432504 17084 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:21:29.441951 17092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:21:29.441965 17091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:21:29.442011 17084 server_base.cc:1034] running on GCE node
W20260618 22:21:29.441991 17094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:21:29.442653 17084 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:21:29.443295 17084 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:21:29.444473 17084 hybrid_clock.cc:648] HybridClock initialized: now 1781821289444463 us; error 32 us; skew 500 ppm
Jun 18 22:21:29 dist-test-slave-g6r4 krb5kdc[15764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821289, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:21:29.447129 17084 init.cc:377] Logged in from keytab as kudu/127.0.98.66@KRBTEST.COM (short username kudu)
I20260618 22:21:29.447975 17084 webserver.cc:460] Webserver started at http://127.0.98.66:39183/ using document root <none> and password file <none>
I20260618 22:21:29.448509 17084 fs_manager.cc:359] Metadata directory not provided
I20260618 22:21:29.448632 17084 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:21:29.452271 17084 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260618 22:21:29.454212 17101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:29.455334 17084 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:21:29.455473 17084 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-1/wal
uuid: "610a350e9c644e61bd41f8646ed9f7c7"
format_stamp: "Formatted at 2026-06-18 22:20:47 on dist-test-slave-g6r4"
server_key: "be8fc677a8c3224a0f362f5a111672d8"
server_key_iv: "9375ad4ed12a38ad90323ab04f36043b"
server_key_version: "encryptionkey@0"
I20260618 22:21:29.456017 17084 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:21:29.476668 17084 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:21:29.479502 17084 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:21:29.479702 17084 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:21:29.480240 17084 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:21:29.480998 17084 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:21:29.481068 17084 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:29.481124 17084 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:21:29.481184 17084 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:29.490123 17084 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:39395
I20260618 22:21:29.490214 17214 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:39395 every 8 connection(s)
I20260618 22:21:29.491070 17084 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
I20260618 22:21:29.494616 17217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:29.492991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39001 (local address 127.0.98.66:35223)
0618 22:21:29.493408 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:29.493876 (+ 468us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":108,"threads_started":1}
W20260618 22:21:29.495074 17215 heartbeater.cc:644] Failed to heartbeat to 127.0.98.126:39001 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.126:39001: Client connection negotiation failed: client connection to 127.0.98.126:39001: connect: Connection refused (error 111)
I20260618 22:21:29.495606 17217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:29.495419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39001 (local address 127.0.98.66:42365)
0618 22:21:29.495517 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:29.495560 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260618 22:21:29.496181 17217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:29.495987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39001 (local address 127.0.98.66:37525)
0618 22:21:29.496086 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:29.496139 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260618 22:21:29.496363 17215 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260618 22:21:29.497727 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 17084
I20260618 22:21:29.498183 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:44555
--local_ip_for_outbound_sockets=127.0.98.67
--tserver_master_addrs=127.0.98.126:39001
--webserver_port=39675
--webserver_interface=127.0.98.67
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:38325
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:21:29.619684 17218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:21:29.619925 17218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:21:29.620000 17218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:21:29.624078 17218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:21:29.624130 17218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:21:29.624226 17218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260618 22:21:29.629695 17218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:38325
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:44555
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=39675
--tserver_master_addrs=127.0.98.126:39001
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17218
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:21:29.630915 17218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:21:29.632249 17218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:21:29.641716 17223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:21:29.641726 17224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:21:29.641870 17218 server_base.cc:1034] running on GCE node
W20260618 22:21:29.642072 17226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:21:29.642463 17218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:21:29.643103 17218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:21:29.644290 17218 hybrid_clock.cc:648] HybridClock initialized: now 1781821289644258 us; error 43 us; skew 500 ppm
Jun 18 22:21:29 dist-test-slave-g6r4 krb5kdc[15764](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821289, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:21:29.647104 17218 init.cc:377] Logged in from keytab as kudu/127.0.98.67@KRBTEST.COM (short username kudu)
I20260618 22:21:29.647953 17218 webserver.cc:460] Webserver started at http://127.0.98.67:39675/ using document root <none> and password file <none>
I20260618 22:21:29.648533 17218 fs_manager.cc:359] Metadata directory not provided
I20260618 22:21:29.648591 17218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:21:29.652206 17218 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:21:29.654305 17233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:29.655499 17218 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:21:29.655663 17218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-2/wal
uuid: "139063e1765148b181db5d13fdf5ff54"
format_stamp: "Formatted at 2026-06-18 22:20:47 on dist-test-slave-g6r4"
server_key: "0482da59d59c1f4c97da2193113e0cc9"
server_key_iv: "2fe7273eae39923eda6ccc37090ead2b"
server_key_version: "encryptionkey@0"
I20260618 22:21:29.656198 17218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:21:29.672539 17218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:21:29.675506 17218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:21:29.675702 17218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:21:29.676258 17218 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:21:29.677019 17218 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:21:29.677073 17218 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:29.677140 17218 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:21:29.677171 17218 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:29.685475 17218 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:44555
I20260618 22:21:29.685523 17346 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:44555 every 8 connection(s)
I20260618 22:21:29.686468 17218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
I20260618 22:21:29.690194 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:29.688458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39001 (local address 127.0.98.67:49205)
0618 22:21:29.688973 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:29.689444 (+ 471us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":178,"threads_started":1}
W20260618 22:21:29.690655 17347 heartbeater.cc:644] Failed to heartbeat to 127.0.98.126:39001 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.126:39001: Client connection negotiation failed: client connection to 127.0.98.126:39001: connect: Connection refused (error 111)
I20260618 22:21:29.691161 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:29.690953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39001 (local address 127.0.98.67:41153)
0618 22:21:29.691056 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:29.691100 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260618 22:21:29.691902 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:29.691645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39001 (local address 127.0.98.67:54863)
0618 22:21:29.691774 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:29.691830 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260618 22:21:29.692124 17347 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260618 22:21:29.693822 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 17218
I20260618 22:21:30.396526 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:30.396124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39001 (local address 127.0.98.65:59729)
0618 22:21:30.396364 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:30.396417 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":70,"threads_started":1}
I20260618 22:21:30.497159 17352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:30.496807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39001 (local address 127.0.98.66:38409)
0618 22:21:30.497015 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:30.497065 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":65,"threads_started":1}
I20260618 22:21:30.693182 17353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:30.692744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39001 (local address 127.0.98.67:32955)
0618 22:21:30.692991 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:30.693075 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":74,"threads_started":1}
I20260618 22:21:31.032704 17354 raft_consensus.cc:482] T 3b7ae90f164d4812924fa1d35f823d2b P ae5c1a50e67f4aed8830a55970810e8e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:21:31.032948 17354 raft_consensus.cc:504] T 3b7ae90f164d4812924fa1d35f823d2b P ae5c1a50e67f4aed8830a55970810e8e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139063e1765148b181db5d13fdf5ff54" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 44555 } } peers { permanent_uuid: "610a350e9c644e61bd41f8646ed9f7c7" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39395 } } peers { permanent_uuid: "ae5c1a50e67f4aed8830a55970810e8e" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33093 } }
I20260618 22:21:31.034118 17354 leader_election.cc:288] T 3b7ae90f164d4812924fa1d35f823d2b P ae5c1a50e67f4aed8830a55970810e8e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 139063e1765148b181db5d13fdf5ff54 (127.0.98.67:44555), 610a350e9c644e61bd41f8646ed9f7c7 (127.0.98.66:39395)
Jun 18 22:21:31 dist-test-slave-g6r4 krb5kdc[15764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821289, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for kudu/127.0.98.67@KRBTEST.COM
Jun 18 22:21:31 dist-test-slave-g6r4 krb5kdc[15764](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821289, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for kudu/127.0.98.66@KRBTEST.COM
I20260618 22:21:31.048539 17356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:31.034356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:39395 (local address 127.0.98.65:46063)
0618 22:21:31.034743 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:31.034757 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:21:31.034833 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:21:31.036688 (+ 1855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:21:31.036694 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:21:31.037231 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:21:31.037696 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:31.037709 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:31.039029 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:31.039033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:31.040041 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:31.040050 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:31.040750 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:31.040754 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:31.040982 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:31.041702 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:21:31.041719 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:21:31.043713 (+ 1994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:21:31.045779 (+ 2066us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:21:31.045781 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:21:31.045783 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:21:31.046134 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:21:31.046772 (+ 638us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:21:31.046777 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:21:31.046781 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:21:31.046868 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:21:31.047445 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:21:31.047451 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:21:31.047772 (+ 321us) client_negotiation.cc:769] Sending connection context
0618 22:21:31.047978 (+ 206us) client_negotiation.cc:240] Negotiation successful
0618 22:21:31.048329 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":322,"mutex_wait_us":125,"spinlock_wait_cycles":10112,"thread_start_us":69,"threads_started":1}
I20260618 22:21:31.048556 17358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:31.034779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:46063 (local address 127.0.98.66:39395)
0618 22:21:31.035078 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:31.035089 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:31.035122 (+ 33us) server_negotiation.cc:407] Connection header received
0618 22:21:31.036071 (+ 949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:31.036078 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:31.036365 (+ 287us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:31.036658 (+ 293us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:21:31.037886 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:31.038891 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:31.040179 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:31.040618 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:31.040784 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:31.043894 (+ 3110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:21:31.043899 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:21:31.043909 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:21:31.043935 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:21:31.045666 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:21:31.046568 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:21:31.046577 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:21:31.046580 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:21:31.046682 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:21:31.046957 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:21:31.046960 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:21:31.046962 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:21:31.047347 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:21:31.047453 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:31.048109 (+ 656us) server_negotiation.cc:299] Negotiation successful
0618 22:21:31.048377 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":112,"threads_started":1}
I20260618 22:21:31.048588 17357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:31.034423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:48939 (local address 127.0.98.67:44555)
0618 22:21:31.034696 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:31.034724 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:31.034767 (+ 43us) server_negotiation.cc:407] Connection header received
0618 22:21:31.035938 (+ 1171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:31.035946 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:31.036296 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:31.036592 (+ 296us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:21:31.037869 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:31.038905 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:31.040155 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:31.040632 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:31.040823 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:31.043600 (+ 2777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:21:31.043606 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:21:31.043622 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:21:31.043660 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:21:31.045507 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:21:31.046255 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:21:31.046259 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:21:31.046264 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:21:31.046420 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:21:31.046818 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:21:31.046820 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:21:31.046821 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:21:31.047310 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:21:31.047450 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:31.048109 (+ 659us) server_negotiation.cc:299] Negotiation successful
0618 22:21:31.048377 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":53,"threads_started":1}
I20260618 22:21:31.048538 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:31.034268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:44555 (local address 127.0.98.65:48939)
0618 22:21:31.034535 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:31.034550 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:21:31.034655 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:21:31.036688 (+ 2033us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:21:31.036694 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:21:31.037231 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:21:31.037695 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:31.037709 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:31.039055 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:31.039057 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:31.040037 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:31.040047 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:31.040786 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:31.040789 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:31.040978 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:31.041742 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:21:31.041753 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:21:31.043365 (+ 1612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:21:31.045626 (+ 2261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:21:31.045631 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:21:31.045644 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:21:31.046132 (+ 488us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:21:31.046539 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:21:31.046542 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:21:31.046544 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:21:31.046702 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:21:31.047445 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:21:31.047451 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:21:31.047772 (+ 321us) client_negotiation.cc:769] Sending connection context
0618 22:21:31.047978 (+ 206us) client_negotiation.cc:240] Negotiation successful
0618 22:21:31.048378 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"mutex_wait_us":397,"spinlock_wait_cycles":46080,"thread_start_us":75,"threads_started":1}
W20260618 22:21:31.051396 16969 leader_election.cc:341] T 3b7ae90f164d4812924fa1d35f823d2b P ae5c1a50e67f4aed8830a55970810e8e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 610a350e9c644e61bd41f8646ed9f7c7 (127.0.98.66:39395): Not found: Tablet not found: 3b7ae90f164d4812924fa1d35f823d2b
W20260618 22:21:31.051666 16967 leader_election.cc:341] T 3b7ae90f164d4812924fa1d35f823d2b P ae5c1a50e67f4aed8830a55970810e8e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 139063e1765148b181db5d13fdf5ff54 (127.0.98.67:44555): Not found: Tablet not found: 3b7ae90f164d4812924fa1d35f823d2b
I20260618 22:21:31.051743 16967 leader_election.cc:302] T 3b7ae90f164d4812924fa1d35f823d2b P ae5c1a50e67f4aed8830a55970810e8e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae5c1a50e67f4aed8830a55970810e8e; no voters: 139063e1765148b181db5d13fdf5ff54, 610a350e9c644e61bd41f8646ed9f7c7
I20260618 22:21:31.052021 17354 raft_consensus.cc:2726] T 3b7ae90f164d4812924fa1d35f823d2b P ae5c1a50e67f4aed8830a55970810e8e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260618 22:21:31.397650 17355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:31.397322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39001 (local address 127.0.98.65:51709)
0618 22:21:31.397463 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:31.397565 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260618 22:21:31.498382 17359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:31.497886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39001 (local address 127.0.98.66:46171)
0618 22:21:31.498184 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:31.498246 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":80,"threads_started":1}
I20260618 22:21:31.694283 17360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:31.693854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39001 (local address 127.0.98.67:41785)
0618 22:21:31.694147 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:31.694201 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":127,"threads_started":1}
I20260618 22:21:32.398845 17361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:32.398457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39001 (local address 127.0.98.65:45155)
0618 22:21:32.398696 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:32.398750 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":71,"threads_started":1}
I20260618 22:21:32.499455 17362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:32.499095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39001 (local address 127.0.98.66:52967)
0618 22:21:32.499313 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:32.499362 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":72,"threads_started":1}
I20260618 22:21:32.695405 17363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:32.695013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39001 (local address 127.0.98.67:51111)
0618 22:21:32.695242 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:32.695298 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":65,"threads_started":1}
I20260618 22:21:32.985113 17364 raft_consensus.cc:482] T 3b7ae90f164d4812924fa1d35f823d2b P ae5c1a50e67f4aed8830a55970810e8e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:21:32.985244 17364 raft_consensus.cc:504] T 3b7ae90f164d4812924fa1d35f823d2b P ae5c1a50e67f4aed8830a55970810e8e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139063e1765148b181db5d13fdf5ff54" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 44555 } } peers { permanent_uuid: "610a350e9c644e61bd41f8646ed9f7c7" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39395 } } peers { permanent_uuid: "ae5c1a50e67f4aed8830a55970810e8e" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33093 } }
I20260618 22:21:32.985666 17364 leader_election.cc:288] T 3b7ae90f164d4812924fa1d35f823d2b P ae5c1a50e67f4aed8830a55970810e8e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 139063e1765148b181db5d13fdf5ff54 (127.0.98.67:44555), 610a350e9c644e61bd41f8646ed9f7c7 (127.0.98.66:39395)
W20260618 22:21:32.986451 16967 leader_election.cc:341] T 3b7ae90f164d4812924fa1d35f823d2b P ae5c1a50e67f4aed8830a55970810e8e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 139063e1765148b181db5d13fdf5ff54 (127.0.98.67:44555): Not found: Tablet not found: 3b7ae90f164d4812924fa1d35f823d2b
W20260618 22:21:32.986536 16969 leader_election.cc:341] T 3b7ae90f164d4812924fa1d35f823d2b P ae5c1a50e67f4aed8830a55970810e8e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 610a350e9c644e61bd41f8646ed9f7c7 (127.0.98.66:39395): Not found: Tablet not found: 3b7ae90f164d4812924fa1d35f823d2b
I20260618 22:21:32.986672 16969 leader_election.cc:302] T 3b7ae90f164d4812924fa1d35f823d2b P ae5c1a50e67f4aed8830a55970810e8e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae5c1a50e67f4aed8830a55970810e8e; no voters: 139063e1765148b181db5d13fdf5ff54, 610a350e9c644e61bd41f8646ed9f7c7
I20260618 22:21:32.986864 17364 raft_consensus.cc:2726] T 3b7ae90f164d4812924fa1d35f823d2b P ae5c1a50e67f4aed8830a55970810e8e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260618 22:21:33.400012 17365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:33.399620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39001 (local address 127.0.98.65:34505)
0618 22:21:33.399865 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:33.399917 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":90,"threads_started":1}
I20260618 22:21:33.500622 17366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:33.500200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39001 (local address 127.0.98.66:36987)
0618 22:21:33.500481 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:33.500536 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":68,"threads_started":1}
I20260618 22:21:33.696458 17367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:33.696085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39001 (local address 127.0.98.67:38851)
0618 22:21:33.696320 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:33.696376 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":69,"threads_started":1}
I20260618 22:21:34.401360 17368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:34.400869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39001 (local address 127.0.98.65:44299)
0618 22:21:34.401164 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:34.401227 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":88,"threads_started":1}
I20260618 22:21:34.501843 17369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:34.501445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39001 (local address 127.0.98.66:44005)
0618 22:21:34.501700 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:34.501757 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":73,"threads_started":1}
I20260618 22:21:34.697523 17370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:34.697170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39001 (local address 127.0.98.67:54647)
0618 22:21:34.697384 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:34.697433 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":68,"threads_started":1}
I20260618 22:21:35.122543 17371 raft_consensus.cc:482] T 3b7ae90f164d4812924fa1d35f823d2b P ae5c1a50e67f4aed8830a55970810e8e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:21:35.122673 17371 raft_consensus.cc:504] T 3b7ae90f164d4812924fa1d35f823d2b P ae5c1a50e67f4aed8830a55970810e8e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139063e1765148b181db5d13fdf5ff54" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 44555 } } peers { permanent_uuid: "610a350e9c644e61bd41f8646ed9f7c7" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39395 } } peers { permanent_uuid: "ae5c1a50e67f4aed8830a55970810e8e" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33093 } }
I20260618 22:21:35.123059 17371 leader_election.cc:288] T 3b7ae90f164d4812924fa1d35f823d2b P ae5c1a50e67f4aed8830a55970810e8e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 139063e1765148b181db5d13fdf5ff54 (127.0.98.67:44555), 610a350e9c644e61bd41f8646ed9f7c7 (127.0.98.66:39395)
W20260618 22:21:35.123816 16967 leader_election.cc:341] T 3b7ae90f164d4812924fa1d35f823d2b P ae5c1a50e67f4aed8830a55970810e8e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 139063e1765148b181db5d13fdf5ff54 (127.0.98.67:44555): Not found: Tablet not found: 3b7ae90f164d4812924fa1d35f823d2b
W20260618 22:21:35.123904 16969 leader_election.cc:341] T 3b7ae90f164d4812924fa1d35f823d2b P ae5c1a50e67f4aed8830a55970810e8e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 610a350e9c644e61bd41f8646ed9f7c7 (127.0.98.66:39395): Not found: Tablet not found: 3b7ae90f164d4812924fa1d35f823d2b
I20260618 22:21:35.124027 16969 leader_election.cc:302] T 3b7ae90f164d4812924fa1d35f823d2b P ae5c1a50e67f4aed8830a55970810e8e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae5c1a50e67f4aed8830a55970810e8e; no voters: 139063e1765148b181db5d13fdf5ff54, 610a350e9c644e61bd41f8646ed9f7c7
I20260618 22:21:35.124161 17371 raft_consensus.cc:2726] T 3b7ae90f164d4812924fa1d35f823d2b P ae5c1a50e67f4aed8830a55970810e8e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260618 22:21:35.402626 17372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:35.402135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39001 (local address 127.0.98.65:38531)
0618 22:21:35.402397 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:35.402456 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":92,"threads_started":1}
I20260618 22:21:35.503113 17373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:35.502761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39001 (local address 127.0.98.66:32975)
0618 22:21:35.502977 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:35.503028 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":69,"threads_started":1}
I20260618 22:21:35.698625 17374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:35.698247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39001 (local address 127.0.98.67:43805)
0618 22:21:35.698482 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:35.698536 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":71,"threads_started":1}
I20260618 22:21:36.403759 17375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:36.403375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39001 (local address 127.0.98.65:42679)
0618 22:21:36.403613 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:36.403666 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":81,"threads_started":1}
I20260618 22:21:36.504166 17376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:36.503738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39001 (local address 127.0.98.66:57875)
0618 22:21:36.504027 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:36.504084 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":70,"threads_started":1}
I20260618 22:21:36.699784 17377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:36.699317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39001 (local address 127.0.98.67:39787)
0618 22:21:36.699644 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:36.699697 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":90,"threads_started":1}
I20260618 22:21:37.404875 17378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:37.404494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39001 (local address 127.0.98.65:56383)
0618 22:21:37.404729 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:37.404782 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":73,"threads_started":1}
I20260618 22:21:37.505405 17379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:37.504937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39001 (local address 127.0.98.66:33719)
0618 22:21:37.505196 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:37.505299 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":71,"threads_started":1}
I20260618 22:21:37.700870 17380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:37.700446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39001 (local address 127.0.98.67:37781)
0618 22:21:37.700714 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:37.700784 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":70,"threads_started":1}
I20260618 22:21:38.406047 17381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:38.405628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39001 (local address 127.0.98.65:38709)
0618 22:21:38.405865 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:38.405952 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":77,"threads_started":1}
I20260618 22:21:38.506662 17382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:38.506277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39001 (local address 127.0.98.66:36769)
0618 22:21:38.506514 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:38.506570 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":75,"threads_started":1}
I20260618 22:21:38.702060 17383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:38.701634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39001 (local address 127.0.98.67:59851)
0618 22:21:38.701866 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:38.701970 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":72,"threads_started":1}
I20260618 22:21:38.749770 17384 raft_consensus.cc:482] T 3b7ae90f164d4812924fa1d35f823d2b P ae5c1a50e67f4aed8830a55970810e8e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:21:38.749979 17384 raft_consensus.cc:504] T 3b7ae90f164d4812924fa1d35f823d2b P ae5c1a50e67f4aed8830a55970810e8e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "139063e1765148b181db5d13fdf5ff54" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 44555 } } peers { permanent_uuid: "610a350e9c644e61bd41f8646ed9f7c7" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 39395 } } peers { permanent_uuid: "ae5c1a50e67f4aed8830a55970810e8e" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33093 } }
I20260618 22:21:38.750422 17384 leader_election.cc:288] T 3b7ae90f164d4812924fa1d35f823d2b P ae5c1a50e67f4aed8830a55970810e8e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 139063e1765148b181db5d13fdf5ff54 (127.0.98.67:44555), 610a350e9c644e61bd41f8646ed9f7c7 (127.0.98.66:39395)
W20260618 22:21:38.751117 16969 leader_election.cc:341] T 3b7ae90f164d4812924fa1d35f823d2b P ae5c1a50e67f4aed8830a55970810e8e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 610a350e9c644e61bd41f8646ed9f7c7 (127.0.98.66:39395): Not found: Tablet not found: 3b7ae90f164d4812924fa1d35f823d2b
W20260618 22:21:38.751353 16967 leader_election.cc:341] T 3b7ae90f164d4812924fa1d35f823d2b P ae5c1a50e67f4aed8830a55970810e8e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 139063e1765148b181db5d13fdf5ff54 (127.0.98.67:44555): Not found: Tablet not found: 3b7ae90f164d4812924fa1d35f823d2b
I20260618 22:21:38.751430 16967 leader_election.cc:302] T 3b7ae90f164d4812924fa1d35f823d2b P ae5c1a50e67f4aed8830a55970810e8e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ae5c1a50e67f4aed8830a55970810e8e; no voters: 139063e1765148b181db5d13fdf5ff54, 610a350e9c644e61bd41f8646ed9f7c7
I20260618 22:21:38.751662 17384 raft_consensus.cc:2726] T 3b7ae90f164d4812924fa1d35f823d2b P ae5c1a50e67f4aed8830a55970810e8e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260618 22:21:39.407281 17385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:39.406903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39001 (local address 127.0.98.65:45619)
0618 22:21:39.407136 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:39.407190 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":72,"threads_started":1}
I20260618 22:21:39.507750 17386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:39.507377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39001 (local address 127.0.98.66:43511)
0618 22:21:39.507611 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:39.507662 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":84,"threads_started":1}
I20260618 22:21:39.703112 17387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:39.702722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39001 (local address 127.0.98.67:48173)
0618 22:21:39.702970 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:39.703023 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":70,"threads_started":1}
I20260618 22:21:40.408439 17388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:40.408040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39001 (local address 127.0.98.65:43629)
0618 22:21:40.408271 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:40.408324 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":73,"threads_started":1}
I20260618 22:21:40.508850 17389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:40.508488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39001 (local address 127.0.98.66:38451)
0618 22:21:40.508710 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:40.508761 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":71,"threads_started":1}
I20260618 22:21:40.704245 17390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:40.703808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39001 (local address 127.0.98.67:46123)
0618 22:21:40.704106 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:40.704161 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":63,"threads_started":1}
I20260618 22:21:41.409615 17391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:41.409216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39001 (local address 127.0.98.65:55121)
0618 22:21:41.409468 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:41.409523 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":80,"threads_started":1}
I20260618 22:21:41.510064 17392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:41.509571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39001 (local address 127.0.98.66:34419)
0618 22:21:41.509838 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:41.509966 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":88,"threads_started":1}
I20260618 22:21:41.705513 17393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:41.705024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39001 (local address 127.0.98.67:53109)
0618 22:21:41.705335 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:41.705420 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":69,"threads_started":1}
I20260618 22:21:42.410842 17394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:42.410437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39001 (local address 127.0.98.65:53125)
0618 22:21:42.410694 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:42.410747 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":89,"threads_started":1}
I20260618 22:21:42.511112 17395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:42.510732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39001 (local address 127.0.98.66:40323)
0618 22:21:42.510963 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:42.511014 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":84,"threads_started":1}
I20260618 22:21:42.706620 17396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:42.706234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39001 (local address 127.0.98.67:52799)
0618 22:21:42.706482 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:42.706530 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":88,"threads_started":1}
I20260618 22:21:43.412021 17397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:43.411634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39001 (local address 127.0.98.65:38389)
0618 22:21:43.411874 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:43.411932 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":76,"threads_started":1}
I20260618 22:21:43.512148 17398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:43.511783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39001 (local address 127.0.98.66:40663)
0618 22:21:43.512007 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:43.512059 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":74,"threads_started":1}
I20260618 22:21:43.707654 17399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:43.707275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39001 (local address 127.0.98.67:54503)
0618 22:21:43.707517 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:43.707568 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":84,"threads_started":1}
I20260618 22:21:44.413466 17400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:44.413071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39001 (local address 127.0.98.65:33099)
0618 22:21:44.413311 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:44.413369 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":71,"threads_started":1}
I20260618 22:21:44.513384 17401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:44.513031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39001 (local address 127.0.98.66:37323)
0618 22:21:44.513252 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:44.513301 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.126:39001: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":68,"threads_started":1}
I20260618 22:21:44.695102 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 16911
I20260618 22:21:44.700682 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 17084
I20260618 22:21:44.705824 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 17218
2026-06-18T22:21:44Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59085 ms)
[----------] 30 tests from SecurityITest (239317 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:21:44 dist-test-slave-g6r4 krb5kdc[17404](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:21:44 dist-test-slave-g6r4 krb5kdc[17404](info): set up 2 sockets
Jun 18 22:21:44 dist-test-slave-g6r4 krb5kdc[17404](info): commencing operation
krb5kdc: starting...
W20260618 22:21:45.745703 393 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/krb5kdc/test-user.keytab.
Jun 18 22:21:45 dist-test-slave-g6r4 krb5kdc[17404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:21:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:21:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260618 22:21:45.864758 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:39823
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:42909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:39823
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:21:45.973755 17417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:21:45.974058 17417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:21:45.974131 17417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:21:45.978039 17417 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:21:45.978096 17417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:21:45.978116 17417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:21:45.978132 17417 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:21:45.978173 17417 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:21:45.983292 17417 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:42909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:39823
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:39823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:21:45.984248 17417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:21:45.985196 17417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:21:45.993677 17423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:21:45.993713 17422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:21:45.993861 17417 server_base.cc:1034] running on GCE node
W20260618 22:21:45.994002 17425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:21:45.994648 17417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:21:45.995669 17417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:21:45.996871 17417 hybrid_clock.cc:648] HybridClock initialized: now 1781821305996865 us; error 29 us; skew 500 ppm
Jun 18 22:21:45 dist-test-slave-g6r4 krb5kdc[17404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821305, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:21:45.999456 17417 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260618 22:21:46.000285 17417 webserver.cc:460] Webserver started at http://127.0.98.126:33345/ using document root <none> and password file <none>
I20260618 22:21:46.000825 17417 fs_manager.cc:359] Metadata directory not provided
I20260618 22:21:46.000913 17417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:21:46.001130 17417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:21:46.003136 17417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/minicluster-data/master-0/data/instance:
uuid: "ef9039f22e8340fc83988c89ed523266"
format_stamp: "Formatted at 2026-06-18 22:21:45 on dist-test-slave-g6r4"
server_key: "63b7f411e775835f327f12e525787316"
server_key_iv: "44fc508dc9da0772f679ec27ac56f4cf"
server_key_version: "encryptionkey@0"
I20260618 22:21:46.003679 17417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/minicluster-data/master-0/wal/instance:
uuid: "ef9039f22e8340fc83988c89ed523266"
format_stamp: "Formatted at 2026-06-18 22:21:45 on dist-test-slave-g6r4"
server_key: "63b7f411e775835f327f12e525787316"
server_key_iv: "44fc508dc9da0772f679ec27ac56f4cf"
server_key_version: "encryptionkey@0"
I20260618 22:21:46.007265 17417 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:21:46.009533 17432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:46.010854 17417 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:21:46.010949 17417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/minicluster-data/master-0/wal
uuid: "ef9039f22e8340fc83988c89ed523266"
format_stamp: "Formatted at 2026-06-18 22:21:45 on dist-test-slave-g6r4"
server_key: "63b7f411e775835f327f12e525787316"
server_key_iv: "44fc508dc9da0772f679ec27ac56f4cf"
server_key_version: "encryptionkey@0"
I20260618 22:21:46.011027 17417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:21:46.024159 17417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:21:46.026775 17417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:21:46.026911 17417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:21:46.033320 17417 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:39823
I20260618 22:21:46.033387 17484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:39823 every 8 connection(s)
I20260618 22:21:46.034220 17417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
I20260618 22:21:46.036785 17485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:21:46.040467 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 17417
I20260618 22:21:46.040593 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/minicluster-data/master-0/wal/instance
I20260618 22:21:46.040886 393 external_mini_cluster.cc:1428] Setting key 499dde3bcd5fa975185538cf0f52593c
I20260618 22:21:46.042528 17485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef9039f22e8340fc83988c89ed523266: Bootstrap starting.
I20260618 22:21:46.045027 17485 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef9039f22e8340fc83988c89ed523266: Neither blocks nor log segments found. Creating new log.
I20260618 22:21:46.045858 17485 log.cc:824] T 00000000000000000000000000000000 P ef9039f22e8340fc83988c89ed523266: Log is configured to *not* fsync() on all Append() calls
I20260618 22:21:46.047536 17485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef9039f22e8340fc83988c89ed523266: No bootstrap required, opened a new log
Jun 18 22:21:46 dist-test-slave-g6r4 krb5kdc[17404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:21:46.050979 17485 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef9039f22e8340fc83988c89ed523266 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef9039f22e8340fc83988c89ed523266" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39823 } }
I20260618 22:21:46.051224 17485 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef9039f22e8340fc83988c89ed523266 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:21:46.051282 17485 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef9039f22e8340fc83988c89ed523266 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef9039f22e8340fc83988c89ed523266, State: Initialized, Role: FOLLOWER
I20260618 22:21:46.051661 17485 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef9039f22e8340fc83988c89ed523266 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef9039f22e8340fc83988c89ed523266" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39823 } }
I20260618 22:21:46.051777 17485 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef9039f22e8340fc83988c89ed523266 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:21:46.051863 17485 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef9039f22e8340fc83988c89ed523266 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:21:46.051985 17485 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef9039f22e8340fc83988c89ed523266 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:21:46.052754 17485 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef9039f22e8340fc83988c89ed523266 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef9039f22e8340fc83988c89ed523266" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39823 } }
I20260618 22:21:46.053071 17485 leader_election.cc:302] T 00000000000000000000000000000000 P ef9039f22e8340fc83988c89ed523266 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef9039f22e8340fc83988c89ed523266; no voters:
I20260618 22:21:46.053364 17485 leader_election.cc:288] T 00000000000000000000000000000000 P ef9039f22e8340fc83988c89ed523266 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:21:46.053486 17490 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef9039f22e8340fc83988c89ed523266 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:21:46.053690 17490 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef9039f22e8340fc83988c89ed523266 [term 1 LEADER]: Becoming Leader. State: Replica: ef9039f22e8340fc83988c89ed523266, State: Running, Role: LEADER
I20260618 22:21:46.054075 17490 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef9039f22e8340fc83988c89ed523266 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef9039f22e8340fc83988c89ed523266" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39823 } }
I20260618 22:21:46.054536 17485 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef9039f22e8340fc83988c89ed523266 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:21:46.055724 17491 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef9039f22e8340fc83988c89ed523266 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef9039f22e8340fc83988c89ed523266" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef9039f22e8340fc83988c89ed523266" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39823 } } }
I20260618 22:21:46.055851 17491 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef9039f22e8340fc83988c89ed523266 [sys.catalog]: This master's current role is: LEADER
I20260618 22:21:46.055864 17488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:46.042244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36556 (local address 127.0.98.126:39823)
0618 22:21:46.042639 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:46.042658 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:46.042699 (+ 41us) server_negotiation.cc:407] Connection header received
0618 22:21:46.043475 (+ 776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:46.043496 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:46.043905 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:46.044235 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:21:46.044879 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:46.046360 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:46.047219 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:46.047731 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:46.047898 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:46.050356 (+ 2458us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:21:46.050362 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:21:46.050383 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:21:46.050409 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:21:46.052290 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:21:46.052708 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:21:46.052713 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:21:46.052719 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:21:46.052878 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:21:46.053140 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:21:46.053144 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:21:46.053146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:21:46.053662 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:21:46.054479 (+ 817us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:46.054620 (+ 141us) server_negotiation.cc:299] Negotiation successful
0618 22:21:46.054902 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":256,"thread_start_us":105,"threads_started":1}
I20260618 22:21:46.056195 17492 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef9039f22e8340fc83988c89ed523266 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef9039f22e8340fc83988c89ed523266. Latest consensus state: current_term: 1 leader_uuid: "ef9039f22e8340fc83988c89ed523266" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef9039f22e8340fc83988c89ed523266" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 39823 } } }
I20260618 22:21:46.056286 17492 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef9039f22e8340fc83988c89ed523266 [sys.catalog]: This master's current role is: LEADER
W20260618 22:21:46.059082 17505 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ef9039f22e8340fc83988c89ed523266: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 22:21:46.059172 17505 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 22:21:46.059257 17500 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:21:46.059918 17500 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:21:46.064690 17500 catalog_manager.cc:1347] Generated new cluster ID: a40b53aa92994b609410ea88610c991e
I20260618 22:21:46.064821 17500 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:21:46.075924 17500 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:21:46.076782 17500 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:21:46.081841 17500 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef9039f22e8340fc83988c89ed523266: Generated new TSK 0
I20260618 22:21:46.082479 17500 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab.
I20260618 22:21:46.113111 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:39823
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:42909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:21:46.219975 17511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:21:46.220202 17511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:21:46.220242 17511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:21:46.224118 17511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:21:46.224166 17511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:21:46.224220 17511 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260618 22:21:46.229168 17511 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:42909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:39823
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17511
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:21:46.230126 17511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:21:46.231035 17511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:21:46.239883 17516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:21:46.239909 17519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:21:46.239894 17517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:21:46.240327 17511 server_base.cc:1034] running on GCE node
I20260618 22:21:46.240649 17511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:21:46.241207 17511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:21:46.242375 17511 hybrid_clock.cc:648] HybridClock initialized: now 1781821306242355 us; error 31 us; skew 500 ppm
Jun 18 22:21:46 dist-test-slave-g6r4 krb5kdc[17404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:21:46.244930 17511 init.cc:377] Logged in from keytab as kudu/127.0.98.65@KRBTEST.COM (short username kudu)
I20260618 22:21:46.245718 17511 webserver.cc:460] Webserver started at http://127.0.98.65:33177/ using document root <none> and password file <none>
I20260618 22:21:46.246325 17511 fs_manager.cc:359] Metadata directory not provided
I20260618 22:21:46.246421 17511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:21:46.246644 17511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:21:46.248633 17511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/minicluster-data/ts-0/data/instance:
uuid: "e1a9199e69ce4a6da7b0fb2cb9b6b1a2"
format_stamp: "Formatted at 2026-06-18 22:21:46 on dist-test-slave-g6r4"
server_key: "2201b52cb8bf26b630b7f71e6bbed728"
server_key_iv: "e22e97e7168803bdb5954777e1be8f23"
server_key_version: "encryptionkey@0"
I20260618 22:21:46.249169 17511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/minicluster-data/ts-0/wal/instance:
uuid: "e1a9199e69ce4a6da7b0fb2cb9b6b1a2"
format_stamp: "Formatted at 2026-06-18 22:21:46 on dist-test-slave-g6r4"
server_key: "2201b52cb8bf26b630b7f71e6bbed728"
server_key_iv: "e22e97e7168803bdb5954777e1be8f23"
server_key_version: "encryptionkey@0"
I20260618 22:21:46.252640 17511 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260618 22:21:46.254961 17526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:46.256189 17511 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:21:46.256319 17511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/minicluster-data/ts-0/wal
uuid: "e1a9199e69ce4a6da7b0fb2cb9b6b1a2"
format_stamp: "Formatted at 2026-06-18 22:21:46 on dist-test-slave-g6r4"
server_key: "2201b52cb8bf26b630b7f71e6bbed728"
server_key_iv: "e22e97e7168803bdb5954777e1be8f23"
server_key_version: "encryptionkey@0"
I20260618 22:21:46.256422 17511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:21:46.287525 17511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:21:46.290288 17511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:21:46.290484 17511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:21:46.291031 17511 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:21:46.291802 17511 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:21:46.291877 17511 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:46.291945 17511 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:21:46.291998 17511 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:46.300833 17511 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:33689
I20260618 22:21:46.300863 17639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:33689 every 8 connection(s)
I20260618 22:21:46.301654 17511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
I20260618 22:21:46.308856 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 17511
I20260618 22:21:46.308995 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/minicluster-data/ts-0/wal/instance
I20260618 22:21:46.309298 393 external_mini_cluster.cc:1428] Setting key 082b9f0692950c9c1a9ddd344194fd02
Jun 18 22:21:46 dist-test-slave-g6r4 krb5kdc[17404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:21:46.314515 17488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:46.303384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:59531 (local address 127.0.98.126:39823)
0618 22:21:46.303505 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:46.303510 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:46.304281 (+ 771us) server_negotiation.cc:407] Connection header received
0618 22:21:46.305088 (+ 807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:46.305092 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:46.305138 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:46.305218 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:21:46.306407 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:46.306919 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:46.307964 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:46.308199 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:46.308277 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:46.310909 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:21:46.310914 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:21:46.310921 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:21:46.310943 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:21:46.312326 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:21:46.312840 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:21:46.312843 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:21:46.312845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:21:46.312925 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:21:46.313271 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:21:46.313274 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:21:46.313276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:21:46.313450 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:21:46.313552 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:46.314201 (+ 649us) server_negotiation.cc:299] Negotiation successful
0618 22:21:46.314324 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260618 22:21:46.315300 17642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:46.303698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39823 (local address 127.0.98.65:59531)
0618 22:21:46.304119 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:46.304159 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0618 22:21:46.304891 (+ 732us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:21:46.305391 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:21:46.305397 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:21:46.305728 (+ 331us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:21:46.306275 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:46.306288 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:46.307020 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:46.307023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:46.307860 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:46.307869 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:46.308320 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:46.308323 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:46.308452 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:46.309051 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:21:46.309066 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:21:46.310750 (+ 1684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:21:46.312458 (+ 1708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:21:46.312464 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:21:46.312477 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:21:46.312729 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:21:46.313025 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:21:46.313028 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:21:46.313030 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:21:46.313163 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:21:46.313563 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:21:46.313570 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:21:46.313935 (+ 365us) client_negotiation.cc:769] Sending connection context
0618 22:21:46.314140 (+ 205us) client_negotiation.cc:240] Negotiation successful
0618 22:21:46.314444 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":110,"threads_started":1}
I20260618 22:21:46.316481 17640 heartbeater.cc:347] Connected to a master server at 127.0.98.126:39823
I20260618 22:21:46.316717 17640 heartbeater.cc:464] Registering TS with master...
I20260618 22:21:46.317180 17640 heartbeater.cc:511] Master 127.0.98.126:39823 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.66@KRBTEST.COM; defaulting to no policy
I20260618 22:21:46.318848 17449 ts_manager.cc:194] Registered new tserver with Master: e1a9199e69ce4a6da7b0fb2cb9b6b1a2 (127.0.98.65:33689)
I20260618 22:21:46.320108 17449 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.65@KRBTEST.COM'} at 127.0.98.65:59531
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab.
I20260618 22:21:46.336314 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:39823
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:42909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:21:46.444430 17645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:21:46.444657 17645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:21:46.444697 17645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:21:46.448747 17645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:21:46.448800 17645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:21:46.448890 17645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260618 22:21:46.454103 17645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:42909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:39823
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:21:46.455138 17645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:21:46.456136 17645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:21:46.465626 17651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:21:46.465658 17653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:21:46.465658 17650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:21:46.465999 17645 server_base.cc:1034] running on GCE node
I20260618 22:21:46.466383 17645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:21:46.467011 17645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:21:46.468194 17645 hybrid_clock.cc:648] HybridClock initialized: now 1781821306468180 us; error 26 us; skew 500 ppm
Jun 18 22:21:46 dist-test-slave-g6r4 krb5kdc[17404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:21:46.470806 17645 init.cc:377] Logged in from keytab as kudu/127.0.98.66@KRBTEST.COM (short username kudu)
I20260618 22:21:46.471643 17645 webserver.cc:460] Webserver started at http://127.0.98.66:39997/ using document root <none> and password file <none>
I20260618 22:21:46.472178 17645 fs_manager.cc:359] Metadata directory not provided
I20260618 22:21:46.472278 17645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:21:46.472499 17645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:21:46.474469 17645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/minicluster-data/ts-1/data/instance:
uuid: "7993d11402b74f788644d55633a0f507"
format_stamp: "Formatted at 2026-06-18 22:21:46 on dist-test-slave-g6r4"
server_key: "13c065cecd1c83f6000239dc04903db0"
server_key_iv: "1e7aa31c538712e4a2bcb02a9a639d80"
server_key_version: "encryptionkey@0"
I20260618 22:21:46.475015 17645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/minicluster-data/ts-1/wal/instance:
uuid: "7993d11402b74f788644d55633a0f507"
format_stamp: "Formatted at 2026-06-18 22:21:46 on dist-test-slave-g6r4"
server_key: "13c065cecd1c83f6000239dc04903db0"
server_key_iv: "1e7aa31c538712e4a2bcb02a9a639d80"
server_key_version: "encryptionkey@0"
I20260618 22:21:46.478567 17645 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260618 22:21:46.480798 17660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:46.482020 17645 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260618 22:21:46.482159 17645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/minicluster-data/ts-1/wal
uuid: "7993d11402b74f788644d55633a0f507"
format_stamp: "Formatted at 2026-06-18 22:21:46 on dist-test-slave-g6r4"
server_key: "13c065cecd1c83f6000239dc04903db0"
server_key_iv: "1e7aa31c538712e4a2bcb02a9a639d80"
server_key_version: "encryptionkey@0"
I20260618 22:21:46.482278 17645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:21:46.500335 17645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:21:46.503118 17645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:21:46.503314 17645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:21:46.503870 17645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:21:46.504639 17645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:21:46.504710 17645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:46.504802 17645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:21:46.504853 17645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:46.513520 17645 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:38251
I20260618 22:21:46.513557 17773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:38251 every 8 connection(s)
I20260618 22:21:46.514475 17645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
I20260618 22:21:46.522053 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 17645
I20260618 22:21:46.522192 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/minicluster-data/ts-1/wal/instance
I20260618 22:21:46.522487 393 external_mini_cluster.cc:1428] Setting key 39ea4fe4e736a9dc2a2813f62eba179a
Jun 18 22:21:46 dist-test-slave-g6r4 krb5kdc[17404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:21:46.527308 17488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:46.516256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:47035 (local address 127.0.98.126:39823)
0618 22:21:46.516382 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:46.516386 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:46.517170 (+ 784us) server_negotiation.cc:407] Connection header received
0618 22:21:46.518092 (+ 922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:46.518096 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:46.518153 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:46.518256 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:21:46.519342 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:46.519891 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:46.521005 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:46.521234 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:46.521346 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:46.523925 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:21:46.523929 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:21:46.523932 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:21:46.523955 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:21:46.525284 (+ 1329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:21:46.525768 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:21:46.525772 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:21:46.525774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:21:46.525856 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:21:46.526223 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:21:46.526228 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:21:46.526230 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:21:46.526407 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:21:46.526514 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:46.527089 (+ 575us) server_negotiation.cc:299] Negotiation successful
0618 22:21:46.527188 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260618 22:21:46.528395 17776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:46.516587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39823 (local address 127.0.98.66:47035)
0618 22:21:46.517019 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:46.517059 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0618 22:21:46.517822 (+ 763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:21:46.518421 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:21:46.518428 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:21:46.518760 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:21:46.519201 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:46.519213 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:46.520014 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:46.520017 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:46.520881 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:46.520889 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:46.521379 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:46.521383 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:46.521541 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:46.522107 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:21:46.522123 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:21:46.523770 (+ 1647us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:21:46.525402 (+ 1632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:21:46.525408 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:21:46.525420 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:21:46.525665 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:21:46.525987 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:21:46.525990 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:21:46.525992 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:21:46.526127 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:21:46.526511 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:21:46.526517 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:21:46.526811 (+ 294us) client_negotiation.cc:769] Sending connection context
0618 22:21:46.527016 (+ 205us) client_negotiation.cc:240] Negotiation successful
0618 22:21:46.527356 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":110,"threads_started":1}
I20260618 22:21:46.529577 17774 heartbeater.cc:347] Connected to a master server at 127.0.98.126:39823
I20260618 22:21:46.529814 17774 heartbeater.cc:464] Registering TS with master...
I20260618 22:21:46.530279 17774 heartbeater.cc:511] Master 127.0.98.126:39823 requested a full tablet report, sending...
I20260618 22:21:46.531265 17449 ts_manager.cc:194] Registered new tserver with Master: 7993d11402b74f788644d55633a0f507 (127.0.98.66:38251)
I20260618 22:21:46.531812 17449 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.66@KRBTEST.COM'} at 127.0.98.66:47035
WARNING: no policy specified for kudu/127.0.98.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab.
I20260618 22:21:46.548719 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:39823
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:42909
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:21:46.656574 17779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:21:46.656812 17779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:21:46.656852 17779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:21:46.660794 17779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:21:46.660843 17779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:21:46.660899 17779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260618 22:21:46.666188 17779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:42909
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:39823
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:21:46.667104 17779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:21:46.668030 17779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:21:46.677273 17784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:21:46.677241 17787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:21:46.677259 17785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:21:46.677479 17779 server_base.cc:1034] running on GCE node
I20260618 22:21:46.678081 17779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:21:46.678687 17779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:21:46.679841 17779 hybrid_clock.cc:648] HybridClock initialized: now 1781821306679811 us; error 43 us; skew 500 ppm
Jun 18 22:21:46 dist-test-slave-g6r4 krb5kdc[17404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:21:46.682525 17779 init.cc:377] Logged in from keytab as kudu/127.0.98.67@KRBTEST.COM (short username kudu)
I20260618 22:21:46.683287 17779 webserver.cc:460] Webserver started at http://127.0.98.67:43803/ using document root <none> and password file <none>
I20260618 22:21:46.683804 17779 fs_manager.cc:359] Metadata directory not provided
I20260618 22:21:46.683861 17779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:21:46.684015 17779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:21:46.685999 17779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/minicluster-data/ts-2/data/instance:
uuid: "5b589802b5b744fb97db186cca9b8086"
format_stamp: "Formatted at 2026-06-18 22:21:46 on dist-test-slave-g6r4"
server_key: "4e5444026fc9a430b4b58b918a556949"
server_key_iv: "6aa064f80ca16b1e76860c6bb2dd7fcb"
server_key_version: "encryptionkey@0"
I20260618 22:21:46.686504 17779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/minicluster-data/ts-2/wal/instance:
uuid: "5b589802b5b744fb97db186cca9b8086"
format_stamp: "Formatted at 2026-06-18 22:21:46 on dist-test-slave-g6r4"
server_key: "4e5444026fc9a430b4b58b918a556949"
server_key_iv: "6aa064f80ca16b1e76860c6bb2dd7fcb"
server_key_version: "encryptionkey@0"
I20260618 22:21:46.690680 17779 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20260618 22:21:46.693130 17794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:46.694335 17779 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.003s
I20260618 22:21:46.694449 17779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/minicluster-data/ts-2/wal
uuid: "5b589802b5b744fb97db186cca9b8086"
format_stamp: "Formatted at 2026-06-18 22:21:46 on dist-test-slave-g6r4"
server_key: "4e5444026fc9a430b4b58b918a556949"
server_key_iv: "6aa064f80ca16b1e76860c6bb2dd7fcb"
server_key_version: "encryptionkey@0"
I20260618 22:21:46.694550 17779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:21:46.702901 17779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:21:46.705425 17779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:21:46.705605 17779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:21:46.706199 17779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:21:46.706941 17779 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:21:46.706986 17779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:46.707050 17779 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:21:46.707088 17779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:46.716192 17779 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:35929
I20260618 22:21:46.716312 17907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:35929 every 8 connection(s)
I20260618 22:21:46.717047 17779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
I20260618 22:21:46.724115 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 17779
I20260618 22:21:46.724270 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/minicluster-data/ts-2/wal/instance
I20260618 22:21:46.724534 393 external_mini_cluster.cc:1428] Setting key 647e6e2845e38e1a9e9fa1bba07f4363
Jun 18 22:21:46 dist-test-slave-g6r4 krb5kdc[17404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821306, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:21:46.729830 17488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:46.718878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:46037 (local address 127.0.98.126:39823)
0618 22:21:46.719049 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:46.719053 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:46.719834 (+ 781us) server_negotiation.cc:407] Connection header received
0618 22:21:46.720785 (+ 951us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:46.720788 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:46.720845 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:46.720953 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:21:46.722134 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:46.722672 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:46.723785 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:46.724008 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:46.724120 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:46.726897 (+ 2777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:21:46.726901 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:21:46.726903 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:21:46.726935 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:21:46.728166 (+ 1231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:21:46.728594 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:21:46.728596 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:21:46.728598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:21:46.728668 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:21:46.728939 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:21:46.728941 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:21:46.728942 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:21:46.729100 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:21:46.729173 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:46.729615 (+ 442us) server_negotiation.cc:299] Negotiation successful
0618 22:21:46.729705 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260618 22:21:46.730662 17910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:46.719207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:39823 (local address 127.0.98.67:46037)
0618 22:21:46.719642 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:46.719679 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0618 22:21:46.720566 (+ 887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:21:46.721150 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:21:46.721157 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:21:46.721490 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:21:46.721996 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:46.722011 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:46.722798 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:46.722808 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:46.723661 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:46.723671 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:46.724158 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:46.724162 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:46.724378 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:46.725002 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:21:46.725022 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:21:46.726742 (+ 1720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:21:46.728266 (+ 1524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:21:46.728270 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:21:46.728280 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:21:46.728511 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:21:46.728742 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:21:46.728744 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:21:46.728745 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:21:46.728864 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:21:46.729187 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:21:46.729191 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:21:46.729388 (+ 197us) client_negotiation.cc:769] Sending connection context
0618 22:21:46.729582 (+ 194us) client_negotiation.cc:240] Negotiation successful
0618 22:21:46.729856 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":121,"threads_started":1}
I20260618 22:21:46.731632 17908 heartbeater.cc:347] Connected to a master server at 127.0.98.126:39823
I20260618 22:21:46.731854 17908 heartbeater.cc:464] Registering TS with master...
I20260618 22:21:46.732298 17908 heartbeater.cc:511] Master 127.0.98.126:39823 requested a full tablet report, sending...
I20260618 22:21:46.733229 17449 ts_manager.cc:194] Registered new tserver with Master: 5b589802b5b744fb97db186cca9b8086 (127.0.98.67:35929)
I20260618 22:21:46.733770 17449 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.67@KRBTEST.COM'} at 127.0.98.67:46037
I20260618 22:21:46.738150 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 22:21:46.746829 17488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:46.739997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36564 (local address 127.0.98.126:39823)
0618 22:21:46.740134 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:46.740140 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:46.740395 (+ 255us) server_negotiation.cc:407] Connection header received
0618 22:21:46.740463 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:46.740466 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:46.740503 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:46.740573 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:21:46.741220 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:46.741716 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:46.742556 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:46.742768 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:46.742848 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:46.744170 (+ 1322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:21:46.744172 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:21:46.744174 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:21:46.744196 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:21:46.745394 (+ 1198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:21:46.745782 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:21:46.745785 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:21:46.745786 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:21:46.745856 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:21:46.746138 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:21:46.746140 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:21:46.746142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:21:46.746349 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:21:46.746448 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:46.746633 (+ 185us) server_negotiation.cc:299] Negotiation successful
0618 22:21:46.746720 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260618 22:21:46.749387 17449 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36564:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 22:21:46.752336 17449 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:21:46.766495 17921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:46.760245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47826 (local address 127.0.98.67:35929)
0618 22:21:46.761099 (+ 854us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:46.761111 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:46.761135 (+ 24us) server_negotiation.cc:407] Connection header received
0618 22:21:46.761218 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:46.761224 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:46.761454 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:46.762048 (+ 594us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:21:46.762200 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:46.763327 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:46.765051 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:46.765786 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:46.765994 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:46.766132 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:46.766243 (+ 111us) server_negotiation.cc:299] Negotiation successful
0618 22:21:46.766333 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":751,"thread_start_us":143,"threads_started":1}
I20260618 22:21:46.766507 17920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:46.760129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:35929 (local address 127.0.0.1:47826)
0618 22:21:46.760594 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:46.760639 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0618 22:21:46.760769 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:21:46.761629 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:21:46.761636 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:21:46.761659 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:21:46.761837 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:46.761844 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:46.763480 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:46.763483 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:46.764929 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:46.764939 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:46.765945 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:46.765948 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:46.766069 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:46.766105 (+ 36us) client_negotiation.cc:769] Sending connection context
0618 22:21:46.766218 (+ 113us) client_negotiation.cc:240] Negotiation successful
0618 22:21:46.766333 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":127,"thread_start_us":91,"threads_started":1}
I20260618 22:21:46.766553 17923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:46.760916 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47852 (local address 127.0.98.65:33689)
0618 22:21:46.761288 (+ 372us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:46.761304 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:46.761636 (+ 332us) server_negotiation.cc:407] Connection header received
0618 22:21:46.761719 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:46.761728 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:46.761951 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:46.762103 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:21:46.762293 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:46.763297 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:46.765152 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:46.765779 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:46.765903 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:46.766075 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:46.766312 (+ 237us) server_negotiation.cc:299] Negotiation successful
0618 22:21:46.766405 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":73,"threads_started":1}
I20260618 22:21:46.766791 17919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:46.760908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:38251 (local address 127.0.0.1:35210)
0618 22:21:46.761682 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:46.761696 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:21:46.761777 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:21:46.762372 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:21:46.762376 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:21:46.762393 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:21:46.762469 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:46.762475 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:46.763825 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:46.763830 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:46.764804 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:46.764814 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:46.765948 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:46.765951 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:46.766463 (+ 512us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:46.766483 (+ 20us) client_negotiation.cc:769] Sending connection context
0618 22:21:46.766550 (+ 67us) client_negotiation.cc:240] Negotiation successful
0618 22:21:46.766631 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":716,"mutex_wait_us":4,"thread_start_us":91,"threads_started":1}
I20260618 22:21:46.767437 17924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:46.761063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35210 (local address 127.0.98.66:38251)
0618 22:21:46.761631 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:46.761646 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:46.761780 (+ 134us) server_negotiation.cc:407] Connection header received
0618 22:21:46.761879 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:46.761883 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:46.762075 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:46.762199 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:21:46.762597 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:46.763678 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:46.764923 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:46.765779 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:46.766975 (+ 1196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:46.767130 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:46.767185 (+ 55us) server_negotiation.cc:299] Negotiation successful
0618 22:21:46.767288 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":459,"thread_start_us":128,"threads_started":1}
I20260618 22:21:46.766995 17922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:46.760251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:33689 (local address 127.0.0.1:47852)
0618 22:21:46.761437 (+ 1186us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:46.761451 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:21:46.761544 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:21:46.762086 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:21:46.762089 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:21:46.762108 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:21:46.762199 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:46.762205 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:46.763433 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:46.763436 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:46.765046 (+ 1610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:46.765052 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:46.766018 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:46.766024 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:46.766144 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:46.766173 (+ 29us) client_negotiation.cc:769] Sending connection context
0618 22:21:46.766278 (+ 105us) client_negotiation.cc:240] Negotiation successful
0618 22:21:46.766354 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1125,"mutex_wait_us":55,"thread_start_us":85,"threads_started":1}
I20260618 22:21:46.770054 17842 tablet_service.cc:1467] Processing CreateTablet for tablet b23a2244e020434dac7ee32c2a01fbf4 (DEFAULT_TABLE table=test-table [id=31b206790c344a60a88a9f57249addb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:21:46.770386 17574 tablet_service.cc:1467] Processing CreateTablet for tablet b23a2244e020434dac7ee32c2a01fbf4 (DEFAULT_TABLE table=test-table [id=31b206790c344a60a88a9f57249addb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:21:46.771199 17842 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b23a2244e020434dac7ee32c2a01fbf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:21:46.771270 17574 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b23a2244e020434dac7ee32c2a01fbf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:21:46.771708 17706 tablet_service.cc:1467] Processing CreateTablet for tablet b23a2244e020434dac7ee32c2a01fbf4 (DEFAULT_TABLE table=test-table [id=31b206790c344a60a88a9f57249addb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:21:46.772449 17706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b23a2244e020434dac7ee32c2a01fbf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:21:46.777890 17925 tablet_bootstrap.cc:492] T b23a2244e020434dac7ee32c2a01fbf4 P 7993d11402b74f788644d55633a0f507: Bootstrap starting.
I20260618 22:21:46.779558 17926 tablet_bootstrap.cc:492] T b23a2244e020434dac7ee32c2a01fbf4 P 5b589802b5b744fb97db186cca9b8086: Bootstrap starting.
I20260618 22:21:46.780635 17925 tablet_bootstrap.cc:654] T b23a2244e020434dac7ee32c2a01fbf4 P 7993d11402b74f788644d55633a0f507: Neither blocks nor log segments found. Creating new log.
I20260618 22:21:46.781541 17925 log.cc:824] T b23a2244e020434dac7ee32c2a01fbf4 P 7993d11402b74f788644d55633a0f507: Log is configured to *not* fsync() on all Append() calls
I20260618 22:21:46.781790 17926 tablet_bootstrap.cc:654] T b23a2244e020434dac7ee32c2a01fbf4 P 5b589802b5b744fb97db186cca9b8086: Neither blocks nor log segments found. Creating new log.
I20260618 22:21:46.782191 17927 tablet_bootstrap.cc:492] T b23a2244e020434dac7ee32c2a01fbf4 P e1a9199e69ce4a6da7b0fb2cb9b6b1a2: Bootstrap starting.
I20260618 22:21:46.782910 17926 log.cc:824] T b23a2244e020434dac7ee32c2a01fbf4 P 5b589802b5b744fb97db186cca9b8086: Log is configured to *not* fsync() on all Append() calls
I20260618 22:21:46.783761 17925 tablet_bootstrap.cc:492] T b23a2244e020434dac7ee32c2a01fbf4 P 7993d11402b74f788644d55633a0f507: No bootstrap required, opened a new log
I20260618 22:21:46.783951 17925 ts_tablet_manager.cc:1397] T b23a2244e020434dac7ee32c2a01fbf4 P 7993d11402b74f788644d55633a0f507: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.002s
I20260618 22:21:46.784240 17927 tablet_bootstrap.cc:654] T b23a2244e020434dac7ee32c2a01fbf4 P e1a9199e69ce4a6da7b0fb2cb9b6b1a2: Neither blocks nor log segments found. Creating new log.
I20260618 22:21:46.784655 17926 tablet_bootstrap.cc:492] T b23a2244e020434dac7ee32c2a01fbf4 P 5b589802b5b744fb97db186cca9b8086: No bootstrap required, opened a new log
I20260618 22:21:46.784924 17926 ts_tablet_manager.cc:1397] T b23a2244e020434dac7ee32c2a01fbf4 P 5b589802b5b744fb97db186cca9b8086: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260618 22:21:46.785398 17927 log.cc:824] T b23a2244e020434dac7ee32c2a01fbf4 P e1a9199e69ce4a6da7b0fb2cb9b6b1a2: Log is configured to *not* fsync() on all Append() calls
I20260618 22:21:46.787398 17927 tablet_bootstrap.cc:492] T b23a2244e020434dac7ee32c2a01fbf4 P e1a9199e69ce4a6da7b0fb2cb9b6b1a2: No bootstrap required, opened a new log
I20260618 22:21:46.787580 17927 ts_tablet_manager.cc:1397] T b23a2244e020434dac7ee32c2a01fbf4 P e1a9199e69ce4a6da7b0fb2cb9b6b1a2: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260618 22:21:46.788074 17925 raft_consensus.cc:348] T b23a2244e020434dac7ee32c2a01fbf4 P 7993d11402b74f788644d55633a0f507 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b589802b5b744fb97db186cca9b8086" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 35929 } } peers { permanent_uuid: "e1a9199e69ce4a6da7b0fb2cb9b6b1a2" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33689 } } peers { permanent_uuid: "7993d11402b74f788644d55633a0f507" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 38251 } }
I20260618 22:21:46.788465 17925 raft_consensus.cc:374] T b23a2244e020434dac7ee32c2a01fbf4 P 7993d11402b74f788644d55633a0f507 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:21:46.788368 17926 raft_consensus.cc:348] T b23a2244e020434dac7ee32c2a01fbf4 P 5b589802b5b744fb97db186cca9b8086 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b589802b5b744fb97db186cca9b8086" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 35929 } } peers { permanent_uuid: "e1a9199e69ce4a6da7b0fb2cb9b6b1a2" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33689 } } peers { permanent_uuid: "7993d11402b74f788644d55633a0f507" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 38251 } }
I20260618 22:21:46.788568 17925 raft_consensus.cc:729] T b23a2244e020434dac7ee32c2a01fbf4 P 7993d11402b74f788644d55633a0f507 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7993d11402b74f788644d55633a0f507, State: Initialized, Role: FOLLOWER
I20260618 22:21:46.788682 17926 raft_consensus.cc:374] T b23a2244e020434dac7ee32c2a01fbf4 P 5b589802b5b744fb97db186cca9b8086 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:21:46.788756 17926 raft_consensus.cc:729] T b23a2244e020434dac7ee32c2a01fbf4 P 5b589802b5b744fb97db186cca9b8086 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b589802b5b744fb97db186cca9b8086, State: Initialized, Role: FOLLOWER
I20260618 22:21:46.789062 17925 consensus_queue.cc:260] T b23a2244e020434dac7ee32c2a01fbf4 P 7993d11402b74f788644d55633a0f507 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b589802b5b744fb97db186cca9b8086" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 35929 } } peers { permanent_uuid: "e1a9199e69ce4a6da7b0fb2cb9b6b1a2" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33689 } } peers { permanent_uuid: "7993d11402b74f788644d55633a0f507" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 38251 } }
I20260618 22:21:46.789165 17926 consensus_queue.cc:260] T b23a2244e020434dac7ee32c2a01fbf4 P 5b589802b5b744fb97db186cca9b8086 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b589802b5b744fb97db186cca9b8086" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 35929 } } peers { permanent_uuid: "e1a9199e69ce4a6da7b0fb2cb9b6b1a2" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33689 } } peers { permanent_uuid: "7993d11402b74f788644d55633a0f507" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 38251 } }
I20260618 22:21:46.789744 17774 heartbeater.cc:503] Master 127.0.98.126:39823 was elected leader, sending a full tablet report...
I20260618 22:21:46.790135 17925 ts_tablet_manager.cc:1428] T b23a2244e020434dac7ee32c2a01fbf4 P 7993d11402b74f788644d55633a0f507: Time spent starting tablet: real 0.006s user 0.006s sys 0.002s
I20260618 22:21:46.790817 17908 heartbeater.cc:503] Master 127.0.98.126:39823 was elected leader, sending a full tablet report...
I20260618 22:21:46.791281 17927 raft_consensus.cc:348] T b23a2244e020434dac7ee32c2a01fbf4 P e1a9199e69ce4a6da7b0fb2cb9b6b1a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b589802b5b744fb97db186cca9b8086" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 35929 } } peers { permanent_uuid: "e1a9199e69ce4a6da7b0fb2cb9b6b1a2" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33689 } } peers { permanent_uuid: "7993d11402b74f788644d55633a0f507" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 38251 } }
I20260618 22:21:46.791497 17927 raft_consensus.cc:374] T b23a2244e020434dac7ee32c2a01fbf4 P e1a9199e69ce4a6da7b0fb2cb9b6b1a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:21:46.791554 17927 raft_consensus.cc:729] T b23a2244e020434dac7ee32c2a01fbf4 P e1a9199e69ce4a6da7b0fb2cb9b6b1a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1a9199e69ce4a6da7b0fb2cb9b6b1a2, State: Initialized, Role: FOLLOWER
I20260618 22:21:46.792006 17927 consensus_queue.cc:260] T b23a2244e020434dac7ee32c2a01fbf4 P e1a9199e69ce4a6da7b0fb2cb9b6b1a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b589802b5b744fb97db186cca9b8086" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 35929 } } peers { permanent_uuid: "e1a9199e69ce4a6da7b0fb2cb9b6b1a2" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33689 } } peers { permanent_uuid: "7993d11402b74f788644d55633a0f507" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 38251 } }
I20260618 22:21:46.792271 17926 ts_tablet_manager.cc:1428] T b23a2244e020434dac7ee32c2a01fbf4 P 5b589802b5b744fb97db186cca9b8086: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260618 22:21:46.792779 17640 heartbeater.cc:503] Master 127.0.98.126:39823 was elected leader, sending a full tablet report...
I20260618 22:21:46.793001 17927 ts_tablet_manager.cc:1428] T b23a2244e020434dac7ee32c2a01fbf4 P e1a9199e69ce4a6da7b0fb2cb9b6b1a2: Time spent starting tablet: real 0.005s user 0.000s sys 0.006s
W20260618 22:21:46.802765 17641 tablet.cc:2369] T b23a2244e020434dac7ee32c2a01fbf4 P e1a9199e69ce4a6da7b0fb2cb9b6b1a2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:21:46.863273 17932 raft_consensus.cc:482] T b23a2244e020434dac7ee32c2a01fbf4 P 5b589802b5b744fb97db186cca9b8086 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:21:46.863456 17932 raft_consensus.cc:504] T b23a2244e020434dac7ee32c2a01fbf4 P 5b589802b5b744fb97db186cca9b8086 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b589802b5b744fb97db186cca9b8086" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 35929 } } peers { permanent_uuid: "e1a9199e69ce4a6da7b0fb2cb9b6b1a2" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33689 } } peers { permanent_uuid: "7993d11402b74f788644d55633a0f507" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 38251 } }
I20260618 22:21:46.864481 17932 leader_election.cc:288] T b23a2244e020434dac7ee32c2a01fbf4 P 5b589802b5b744fb97db186cca9b8086 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e1a9199e69ce4a6da7b0fb2cb9b6b1a2 (127.0.98.65:33689), 7993d11402b74f788644d55633a0f507 (127.0.98.66:38251)
I20260618 22:21:46.869328 17924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:46.864968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:43645 (local address 127.0.98.66:38251)
0618 22:21:46.865117 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:46.865124 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:46.865372 (+ 248us) server_negotiation.cc:407] Connection header received
0618 22:21:46.865507 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:46.865509 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:46.865546 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:46.865645 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:21:46.865866 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:46.866589 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:46.868092 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:46.868703 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:46.868815 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:46.868876 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:46.869120 (+ 244us) server_negotiation.cc:299] Negotiation successful
0618 22:21:46.869193 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260618 22:21:46.869349 17910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:46.864746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:33689 (local address 127.0.98.67:49455)
0618 22:21:46.864881 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:46.864898 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0618 22:21:46.864997 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:21:46.865327 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:21:46.865334 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:21:46.865364 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:21:46.865498 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:46.865504 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:46.866539 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:46.866542 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:46.867940 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:46.867946 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:46.868826 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:46.868830 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:46.868973 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:46.869017 (+ 44us) client_negotiation.cc:769] Sending connection context
0618 22:21:46.869089 (+ 72us) client_negotiation.cc:240] Negotiation successful
0618 22:21:46.869192 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":142}
I20260618 22:21:46.869369 17923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:46.864844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:49455 (local address 127.0.98.65:33689)
0618 22:21:46.864970 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:46.864974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:46.864988 (+ 14us) server_negotiation.cc:407] Connection header received
0618 22:21:46.865144 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:46.865147 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:46.865210 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:46.865280 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:21:46.865618 (+ 338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:46.866424 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:46.868104 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:46.868703 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:46.868815 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:46.868883 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:46.869120 (+ 237us) server_negotiation.cc:299] Negotiation successful
0618 22:21:46.869193 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260618 22:21:46.869361 17934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:46.864746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:38251 (local address 127.0.98.67:43645)
0618 22:21:46.865295 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:46.865309 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:21:46.865403 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:21:46.865661 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:21:46.865664 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:21:46.865678 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:21:46.865767 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:46.865771 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:46.866675 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:46.866678 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:46.867936 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:46.867946 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:46.868826 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:46.868830 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:46.868969 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:46.869012 (+ 43us) client_negotiation.cc:769] Sending connection context
0618 22:21:46.869089 (+ 77us) client_negotiation.cc:240] Negotiation successful
0618 22:21:46.869189 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"mutex_wait_us":200,"thread_start_us":184,"threads_started":1}
I20260618 22:21:46.870095 17728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b23a2244e020434dac7ee32c2a01fbf4" candidate_uuid: "5b589802b5b744fb97db186cca9b8086" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7993d11402b74f788644d55633a0f507" is_pre_election: true
I20260618 22:21:46.870153 17594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b23a2244e020434dac7ee32c2a01fbf4" candidate_uuid: "5b589802b5b744fb97db186cca9b8086" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a9199e69ce4a6da7b0fb2cb9b6b1a2" is_pre_election: true
I20260618 22:21:46.870419 17728 raft_consensus.cc:2454] T b23a2244e020434dac7ee32c2a01fbf4 P 7993d11402b74f788644d55633a0f507 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b589802b5b744fb97db186cca9b8086 in term 0.
I20260618 22:21:46.870479 17594 raft_consensus.cc:2454] T b23a2244e020434dac7ee32c2a01fbf4 P e1a9199e69ce4a6da7b0fb2cb9b6b1a2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b589802b5b744fb97db186cca9b8086 in term 0.
I20260618 22:21:46.870951 17797 leader_election.cc:302] T b23a2244e020434dac7ee32c2a01fbf4 P 5b589802b5b744fb97db186cca9b8086 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5b589802b5b744fb97db186cca9b8086, 7993d11402b74f788644d55633a0f507, e1a9199e69ce4a6da7b0fb2cb9b6b1a2; no voters:
I20260618 22:21:46.871209 17932 raft_consensus.cc:2781] T b23a2244e020434dac7ee32c2a01fbf4 P 5b589802b5b744fb97db186cca9b8086 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:21:46.871299 17932 raft_consensus.cc:482] T b23a2244e020434dac7ee32c2a01fbf4 P 5b589802b5b744fb97db186cca9b8086 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:21:46.871343 17932 raft_consensus.cc:3037] T b23a2244e020434dac7ee32c2a01fbf4 P 5b589802b5b744fb97db186cca9b8086 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:21:46.872349 17932 raft_consensus.cc:504] T b23a2244e020434dac7ee32c2a01fbf4 P 5b589802b5b744fb97db186cca9b8086 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b589802b5b744fb97db186cca9b8086" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 35929 } } peers { permanent_uuid: "e1a9199e69ce4a6da7b0fb2cb9b6b1a2" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33689 } } peers { permanent_uuid: "7993d11402b74f788644d55633a0f507" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 38251 } }
I20260618 22:21:46.872681 17932 leader_election.cc:288] T b23a2244e020434dac7ee32c2a01fbf4 P 5b589802b5b744fb97db186cca9b8086 [CANDIDATE]: Term 1 election: Requested vote from peers e1a9199e69ce4a6da7b0fb2cb9b6b1a2 (127.0.98.65:33689), 7993d11402b74f788644d55633a0f507 (127.0.98.66:38251)
I20260618 22:21:46.873044 17594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b23a2244e020434dac7ee32c2a01fbf4" candidate_uuid: "5b589802b5b744fb97db186cca9b8086" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e1a9199e69ce4a6da7b0fb2cb9b6b1a2"
I20260618 22:21:46.873070 17728 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b23a2244e020434dac7ee32c2a01fbf4" candidate_uuid: "5b589802b5b744fb97db186cca9b8086" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7993d11402b74f788644d55633a0f507"
I20260618 22:21:46.873169 17594 raft_consensus.cc:3037] T b23a2244e020434dac7ee32c2a01fbf4 P e1a9199e69ce4a6da7b0fb2cb9b6b1a2 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:21:46.873188 17728 raft_consensus.cc:3037] T b23a2244e020434dac7ee32c2a01fbf4 P 7993d11402b74f788644d55633a0f507 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:21:46.874447 17728 raft_consensus.cc:2454] T b23a2244e020434dac7ee32c2a01fbf4 P 7993d11402b74f788644d55633a0f507 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b589802b5b744fb97db186cca9b8086 in term 1.
I20260618 22:21:46.874447 17594 raft_consensus.cc:2454] T b23a2244e020434dac7ee32c2a01fbf4 P e1a9199e69ce4a6da7b0fb2cb9b6b1a2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b589802b5b744fb97db186cca9b8086 in term 1.
I20260618 22:21:46.874773 17796 leader_election.cc:302] T b23a2244e020434dac7ee32c2a01fbf4 P 5b589802b5b744fb97db186cca9b8086 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b589802b5b744fb97db186cca9b8086, e1a9199e69ce4a6da7b0fb2cb9b6b1a2; no voters:
I20260618 22:21:46.874934 17932 raft_consensus.cc:2781] T b23a2244e020434dac7ee32c2a01fbf4 P 5b589802b5b744fb97db186cca9b8086 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:21:46.875149 17932 raft_consensus.cc:686] T b23a2244e020434dac7ee32c2a01fbf4 P 5b589802b5b744fb97db186cca9b8086 [term 1 LEADER]: Becoming Leader. State: Replica: 5b589802b5b744fb97db186cca9b8086, State: Running, Role: LEADER
I20260618 22:21:46.875460 17932 consensus_queue.cc:237] T b23a2244e020434dac7ee32c2a01fbf4 P 5b589802b5b744fb97db186cca9b8086 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b589802b5b744fb97db186cca9b8086" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 35929 } } peers { permanent_uuid: "e1a9199e69ce4a6da7b0fb2cb9b6b1a2" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33689 } } peers { permanent_uuid: "7993d11402b74f788644d55633a0f507" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 38251 } }
I20260618 22:21:46.878528 17447 catalog_manager.cc:5495] T b23a2244e020434dac7ee32c2a01fbf4 P 5b589802b5b744fb97db186cca9b8086 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b589802b5b744fb97db186cca9b8086 (127.0.98.67). New cstate: current_term: 1 leader_uuid: "5b589802b5b744fb97db186cca9b8086" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b589802b5b744fb97db186cca9b8086" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 35929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e1a9199e69ce4a6da7b0fb2cb9b6b1a2" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33689 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7993d11402b74f788644d55633a0f507" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 38251 } health_report { overall_health: UNKNOWN } } }
I20260618 22:21:46.916169 17921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:46.912510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47842 (local address 127.0.98.67:35929)
0618 22:21:46.912650 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:46.912654 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:46.912669 (+ 15us) server_negotiation.cc:407] Connection header received
0618 22:21:46.912844 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:46.912853 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:46.912910 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:46.913020 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:21:46.913256 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:46.913757 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:46.914690 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:46.914927 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:46.915039 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:46.915344 (+ 305us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:21:46.915769 (+ 425us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:21:46.915909 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:46.915945 (+ 36us) server_negotiation.cc:299] Negotiation successful
0618 22:21:46.915999 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260618 22:21:46.921733 17594 raft_consensus.cc:1261] T b23a2244e020434dac7ee32c2a01fbf4 P e1a9199e69ce4a6da7b0fb2cb9b6b1a2 [term 1 FOLLOWER]: Refusing update from remote peer 5b589802b5b744fb97db186cca9b8086: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:21:46.921737 17728 raft_consensus.cc:1261] T b23a2244e020434dac7ee32c2a01fbf4 P 7993d11402b74f788644d55633a0f507 [term 1 FOLLOWER]: Refusing update from remote peer 5b589802b5b744fb97db186cca9b8086: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:21:46.922501 17932 consensus_queue.cc:1035] T b23a2244e020434dac7ee32c2a01fbf4 P 5b589802b5b744fb97db186cca9b8086 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7993d11402b74f788644d55633a0f507" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 38251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:21:46.922724 17935 consensus_queue.cc:1035] T b23a2244e020434dac7ee32c2a01fbf4 P 5b589802b5b744fb97db186cca9b8086 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e1a9199e69ce4a6da7b0fb2cb9b6b1a2" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33689 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:21:46.930222 17939 mvcc.cc:205] Tried to move back new op lower bound from 7298340073144483840 to 7298340072964562944. Current Snapshot: MvccSnapshot[applied={T|T < 7298340073144483840}]
I20260618 22:21:46.937593 17448 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:21:46.937733 17448 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:21:46.940095 17448 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ef9039f22e8340fc83988c89ed523266: Sending DeleteTablet for 3 replicas of tablet b23a2244e020434dac7ee32c2a01fbf4
I20260618 22:21:46.940784 17574 tablet_service.cc:1514] Processing DeleteTablet for tablet b23a2244e020434dac7ee32c2a01fbf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:21:46 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:47852
I20260618 22:21:46.940973 17842 tablet_service.cc:1514] Processing DeleteTablet for tablet b23a2244e020434dac7ee32c2a01fbf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:21:46 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:47826
I20260618 22:21:46.941284 17706 tablet_service.cc:1514] Processing DeleteTablet for tablet b23a2244e020434dac7ee32c2a01fbf4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:21:46 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:35210
I20260618 22:21:46.941439 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 17511
I20260618 22:21:46.942229 17950 tablet_replica.cc:330] T b23a2244e020434dac7ee32c2a01fbf4 P 7993d11402b74f788644d55633a0f507: stopping tablet replica
I20260618 22:21:46.942560 17950 raft_consensus.cc:2229] T b23a2244e020434dac7ee32c2a01fbf4 P 7993d11402b74f788644d55633a0f507 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:21:46.942951 17950 raft_consensus.cc:2258] T b23a2244e020434dac7ee32c2a01fbf4 P 7993d11402b74f788644d55633a0f507 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:21:46.943537 17949 tablet_replica.cc:330] T b23a2244e020434dac7ee32c2a01fbf4 P 5b589802b5b744fb97db186cca9b8086: stopping tablet replica
I20260618 22:21:46.943895 17949 raft_consensus.cc:2229] T b23a2244e020434dac7ee32c2a01fbf4 P 5b589802b5b744fb97db186cca9b8086 [term 1 LEADER]: Raft consensus shutting down.
I20260618 22:21:46.944258 17949 raft_consensus.cc:2258] T b23a2244e020434dac7ee32c2a01fbf4 P 5b589802b5b744fb97db186cca9b8086 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:21:46.944514 17950 ts_tablet_manager.cc:1905] T b23a2244e020434dac7ee32c2a01fbf4 P 7993d11402b74f788644d55633a0f507: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:21:46.945664 17949 ts_tablet_manager.cc:1905] T b23a2244e020434dac7ee32c2a01fbf4 P 5b589802b5b744fb97db186cca9b8086: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:21:46.947299 17950 ts_tablet_manager.cc:1918] T b23a2244e020434dac7ee32c2a01fbf4 P 7993d11402b74f788644d55633a0f507: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:21:46.947470 17950 log.cc:1196] T b23a2244e020434dac7ee32c2a01fbf4 P 7993d11402b74f788644d55633a0f507: Deleting WAL directory at /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/minicluster-data/ts-1/wal/wals/b23a2244e020434dac7ee32c2a01fbf4
I20260618 22:21:46.947888 17950 ts_tablet_manager.cc:1939] T b23a2244e020434dac7ee32c2a01fbf4 P 7993d11402b74f788644d55633a0f507: Deleting consensus metadata
I20260618 22:21:46.948488 17949 ts_tablet_manager.cc:1918] T b23a2244e020434dac7ee32c2a01fbf4 P 5b589802b5b744fb97db186cca9b8086: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:21:46.948654 17949 log.cc:1196] T b23a2244e020434dac7ee32c2a01fbf4 P 5b589802b5b744fb97db186cca9b8086: Deleting WAL directory at /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781821065380026-393-0/minicluster-data/ts-2/wal/wals/b23a2244e020434dac7ee32c2a01fbf4
W20260618 22:21:46.948868 17434 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.65:33689 (error 108)
I20260618 22:21:46.949038 17949 ts_tablet_manager.cc:1939] T b23a2244e020434dac7ee32c2a01fbf4 P 5b589802b5b744fb97db186cca9b8086: Deleting consensus metadata
I20260618 22:21:46.949432 17435 catalog_manager.cc:4841] TS 7993d11402b74f788644d55633a0f507 (127.0.98.66:38251): tablet b23a2244e020434dac7ee32c2a01fbf4 (table test-table [id=31b206790c344a60a88a9f57249addb6]) successfully deleted
I20260618 22:21:46.949982 17435 catalog_manager.cc:4841] TS 5b589802b5b744fb97db186cca9b8086 (127.0.98.67:35929): tablet b23a2244e020434dac7ee32c2a01fbf4 (table test-table [id=31b206790c344a60a88a9f57249addb6]) successfully deleted
I20260618 22:21:46.950022 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 17645
I20260618 22:21:46.954803 17922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:46.954453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:33689 (local address 127.0.0.1:47858)
0618 22:21:46.954614 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:46.954697 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.65:33689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260618 22:21:46.955025 17434 catalog_manager.cc:4584] TS e1a9199e69ce4a6da7b0fb2cb9b6b1a2 (127.0.98.65:33689): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b23a2244e020434dac7ee32c2a01fbf4: Network error: Client connection negotiation failed: client connection to 127.0.98.65:33689: connect: Connection refused (error 111)
I20260618 22:21:46.955964 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 17779
I20260618 22:21:46.961788 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 17417
2026-06-18T22:21:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2256 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-18T22:21:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:21:46Z Disabled control of system clock
I20260618 22:21:46.991863 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821065380026-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821065380026-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821065380026-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:40847
--webserver_interface=127.0.98.126
--webserver_port=0
--builtin_ntp_servers=127.0.98.84:44135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:40847
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260618 22:21:47.099195 17955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:21:47.099444 17955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:21:47.099520 17955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:21:47.103458 17955 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:21:47.103508 17955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:21:47.103528 17955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:21:47.103588 17955 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:21:47.103623 17955 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:21:47.108425 17955 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:44135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821065380026-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821065380026-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:40847
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:40847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821065380026-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:21:47.109432 17955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:21:47.110776 17955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:21:47.118762 17960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:21:47.118930 17955 server_base.cc:1034] running on GCE node
W20260618 22:21:47.118789 17963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:21:47.119098 17961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:21:47.119585 17955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:21:47.120657 17955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:21:47.121870 17955 hybrid_clock.cc:648] HybridClock initialized: now 1781821307121849 us; error 36 us; skew 500 ppm
I20260618 22:21:47.123826 17955 webserver.cc:460] Webserver started at http://127.0.98.126:46503/ using document root <none> and password file <none>
I20260618 22:21:47.124387 17955 fs_manager.cc:359] Metadata directory not provided
I20260618 22:21:47.124482 17955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:21:47.124712 17955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:21:47.126832 17955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821065380026-393-0/minicluster-data/master-0/data/instance:
uuid: "dbcdede77e4d40a8aafa6e1c9c933f98"
format_stamp: "Formatted at 2026-06-18 22:21:47 on dist-test-slave-g6r4"
server_key: "7e0f909e297977bbc25dfd2e63cea1ce"
server_key_iv: "64c6b6dfc88556141dd11b70e66daec8"
server_key_version: "encryptionkey@0"
I20260618 22:21:47.127414 17955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821065380026-393-0/minicluster-data/master-0/wal/instance:
uuid: "dbcdede77e4d40a8aafa6e1c9c933f98"
format_stamp: "Formatted at 2026-06-18 22:21:47 on dist-test-slave-g6r4"
server_key: "7e0f909e297977bbc25dfd2e63cea1ce"
server_key_iv: "64c6b6dfc88556141dd11b70e66daec8"
server_key_version: "encryptionkey@0"
I20260618 22:21:47.131039 17955 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:21:47.133423 17969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:47.134624 17955 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260618 22:21:47.134759 17955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821065380026-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821065380026-393-0/minicluster-data/master-0/wal
uuid: "dbcdede77e4d40a8aafa6e1c9c933f98"
format_stamp: "Formatted at 2026-06-18 22:21:47 on dist-test-slave-g6r4"
server_key: "7e0f909e297977bbc25dfd2e63cea1ce"
server_key_iv: "64c6b6dfc88556141dd11b70e66daec8"
server_key_version: "encryptionkey@0"
I20260618 22:21:47.134866 17955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821065380026-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821065380026-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821065380026-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:21:47.151700 17955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:21:47.152437 17955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:21:47.152652 17955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:21:47.159930 17955 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:40847
I20260618 22:21:47.159950 18021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:40847 every 8 connection(s)
I20260618 22:21:47.160843 17955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
I20260618 22:21:47.163890 18022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:21:47.167162 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 17955
I20260618 22:21:47.167291 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821065380026-393-0/minicluster-data/master-0/wal/instance
I20260618 22:21:47.167569 393 external_mini_cluster.cc:1428] Setting key 5425bab403535d91e877d70449e48be4
I20260618 22:21:47.169617 18022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbcdede77e4d40a8aafa6e1c9c933f98: Bootstrap starting.
I20260618 22:21:47.171691 18022 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dbcdede77e4d40a8aafa6e1c9c933f98: Neither blocks nor log segments found. Creating new log.
I20260618 22:21:47.172380 18022 log.cc:824] T 00000000000000000000000000000000 P dbcdede77e4d40a8aafa6e1c9c933f98: Log is configured to *not* fsync() on all Append() calls
I20260618 22:21:47.174414 18022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbcdede77e4d40a8aafa6e1c9c933f98: No bootstrap required, opened a new log
I20260618 22:21:47.176913 18025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:47.168950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54070 (local address 127.0.98.126:40847)
0618 22:21:47.169565 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:47.169579 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:47.169613 (+ 34us) server_negotiation.cc:407] Connection header received
0618 22:21:47.170318 (+ 705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:47.170335 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:47.170701 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:47.171023 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:21:47.171449 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:47.172446 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:47.173634 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:47.174149 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:47.174286 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:47.175114 (+ 828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:21:47.175119 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:21:47.175131 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:21:47.175512 (+ 381us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:21:47.175562 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:21:47.175581 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:21:47.175654 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:47.175816 (+ 162us) server_negotiation.cc:299] Negotiation successful
0618 22:21:47.176088 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":135,"threads_started":1}
I20260618 22:21:47.177657 18022 raft_consensus.cc:348] T 00000000000000000000000000000000 P dbcdede77e4d40a8aafa6e1c9c933f98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbcdede77e4d40a8aafa6e1c9c933f98" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40847 } }
I20260618 22:21:47.177920 18022 raft_consensus.cc:374] T 00000000000000000000000000000000 P dbcdede77e4d40a8aafa6e1c9c933f98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:21:47.178004 18022 raft_consensus.cc:729] T 00000000000000000000000000000000 P dbcdede77e4d40a8aafa6e1c9c933f98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbcdede77e4d40a8aafa6e1c9c933f98, State: Initialized, Role: FOLLOWER
I20260618 22:21:47.178431 18022 consensus_queue.cc:260] T 00000000000000000000000000000000 P dbcdede77e4d40a8aafa6e1c9c933f98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbcdede77e4d40a8aafa6e1c9c933f98" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40847 } }
I20260618 22:21:47.178553 18022 raft_consensus.cc:388] T 00000000000000000000000000000000 P dbcdede77e4d40a8aafa6e1c9c933f98 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:21:47.178632 18022 raft_consensus.cc:482] T 00000000000000000000000000000000 P dbcdede77e4d40a8aafa6e1c9c933f98 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:21:47.178725 18022 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dbcdede77e4d40a8aafa6e1c9c933f98 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:21:47.179636 18022 raft_consensus.cc:504] T 00000000000000000000000000000000 P dbcdede77e4d40a8aafa6e1c9c933f98 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbcdede77e4d40a8aafa6e1c9c933f98" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40847 } }
I20260618 22:21:47.179950 18022 leader_election.cc:302] T 00000000000000000000000000000000 P dbcdede77e4d40a8aafa6e1c9c933f98 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dbcdede77e4d40a8aafa6e1c9c933f98; no voters:
I20260618 22:21:47.180229 18022 leader_election.cc:288] T 00000000000000000000000000000000 P dbcdede77e4d40a8aafa6e1c9c933f98 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:21:47.180311 18027 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dbcdede77e4d40a8aafa6e1c9c933f98 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:21:47.180574 18027 raft_consensus.cc:686] T 00000000000000000000000000000000 P dbcdede77e4d40a8aafa6e1c9c933f98 [term 1 LEADER]: Becoming Leader. State: Replica: dbcdede77e4d40a8aafa6e1c9c933f98, State: Running, Role: LEADER
I20260618 22:21:47.180935 18027 consensus_queue.cc:237] T 00000000000000000000000000000000 P dbcdede77e4d40a8aafa6e1c9c933f98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbcdede77e4d40a8aafa6e1c9c933f98" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40847 } }
I20260618 22:21:47.181339 18022 sys_catalog.cc:564] T 00000000000000000000000000000000 P dbcdede77e4d40a8aafa6e1c9c933f98 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:21:47.182478 18029 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbcdede77e4d40a8aafa6e1c9c933f98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dbcdede77e4d40a8aafa6e1c9c933f98. Latest consensus state: current_term: 1 leader_uuid: "dbcdede77e4d40a8aafa6e1c9c933f98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbcdede77e4d40a8aafa6e1c9c933f98" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40847 } } }
I20260618 22:21:47.182603 18029 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbcdede77e4d40a8aafa6e1c9c933f98 [sys.catalog]: This master's current role is: LEADER
I20260618 22:21:47.182977 18036 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:21:47.182960 18028 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbcdede77e4d40a8aafa6e1c9c933f98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dbcdede77e4d40a8aafa6e1c9c933f98" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbcdede77e4d40a8aafa6e1c9c933f98" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 40847 } } }
I20260618 22:21:47.183116 18028 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbcdede77e4d40a8aafa6e1c9c933f98 [sys.catalog]: This master's current role is: LEADER
I20260618 22:21:47.186108 18036 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:21:47.191108 18036 catalog_manager.cc:1347] Generated new cluster ID: e06d16cd80a5472cb7da0f2346eb7877
I20260618 22:21:47.191179 18036 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:21:47.201050 18036 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:21:47.201843 18036 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:21:47.208292 18036 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dbcdede77e4d40a8aafa6e1c9c933f98: Generated new TSK 0
I20260618 22:21:47.208897 18036 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:21:47.217466 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821065380026-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821065380026-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821065380026-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:40847
--builtin_ntp_servers=127.0.98.84:44135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260618 22:21:47.339246 18046 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:21:47.339490 18046 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:21:47.339552 18046 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:21:47.343505 18046 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:21:47.343557 18046 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:21:47.343642 18046 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260618 22:21:47.348286 18046 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:44135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821065380026-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821065380026-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:40847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18046
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821065380026-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:21:47.349262 18046 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:21:47.350659 18046 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:21:47.359922 18052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:21:47.359923 18051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:21:47.359956 18046 server_base.cc:1034] running on GCE node
W20260618 22:21:47.359923 18054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:21:47.360538 18046 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:21:47.361176 18046 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:21:47.362339 18046 hybrid_clock.cc:648] HybridClock initialized: now 1781821307362326 us; error 25 us; skew 500 ppm
I20260618 22:21:47.364142 18046 webserver.cc:460] Webserver started at http://127.0.98.65:45973/ using document root <none> and password file <none>
I20260618 22:21:47.364907 18046 fs_manager.cc:359] Metadata directory not provided
I20260618 22:21:47.364970 18046 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:21:47.365238 18046 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:21:47.367317 18046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821065380026-393-0/minicluster-data/ts-0/data/instance:
uuid: "d00f8d610fa74810981fc2c8c0ad2c7f"
format_stamp: "Formatted at 2026-06-18 22:21:47 on dist-test-slave-g6r4"
server_key: "34e753e8c8b4973116ecec4e568d7879"
server_key_iv: "3c39385979e2af76776043e88156fa3c"
server_key_version: "encryptionkey@0"
I20260618 22:21:47.367858 18046 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821065380026-393-0/minicluster-data/ts-0/wal/instance:
uuid: "d00f8d610fa74810981fc2c8c0ad2c7f"
format_stamp: "Formatted at 2026-06-18 22:21:47 on dist-test-slave-g6r4"
server_key: "34e753e8c8b4973116ecec4e568d7879"
server_key_iv: "3c39385979e2af76776043e88156fa3c"
server_key_version: "encryptionkey@0"
I20260618 22:21:47.371395 18046 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260618 22:21:47.373669 18060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:47.374850 18046 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:21:47.374980 18046 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821065380026-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821065380026-393-0/minicluster-data/ts-0/wal
uuid: "d00f8d610fa74810981fc2c8c0ad2c7f"
format_stamp: "Formatted at 2026-06-18 22:21:47 on dist-test-slave-g6r4"
server_key: "34e753e8c8b4973116ecec4e568d7879"
server_key_iv: "3c39385979e2af76776043e88156fa3c"
server_key_version: "encryptionkey@0"
I20260618 22:21:47.375087 18046 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821065380026-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821065380026-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821065380026-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:21:47.394444 18046 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:21:47.395102 18046 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:21:47.395300 18046 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:21:47.395855 18046 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:21:47.396764 18046 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:21:47.396834 18046 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:47.396901 18046 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:21:47.396953 18046 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:47.406181 18046 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:38215
I20260618 22:21:47.406279 18173 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:38215 every 8 connection(s)
I20260618 22:21:47.407106 18046 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
I20260618 22:21:47.414182 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 18046
I20260618 22:21:47.414305 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821065380026-393-0/minicluster-data/ts-0/wal/instance
I20260618 22:21:47.414583 393 external_mini_cluster.cc:1428] Setting key 1ecd79c2e29ebd1b3cc6c6647ca75253
I20260618 22:21:47.416501 18025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:47.408946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:43423 (local address 127.0.98.126:40847)
0618 22:21:47.409117 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:47.409121 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:47.410187 (+ 1066us) server_negotiation.cc:407] Connection header received
0618 22:21:47.411552 (+ 1365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:47.411555 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:47.411609 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:47.411712 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:21:47.412570 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:47.413106 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:47.414359 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:47.414650 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:47.414772 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:47.416016 (+ 1244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:21:47.416019 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:21:47.416021 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:21:47.416032 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:21:47.416049 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:21:47.416056 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:21:47.416142 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:47.416312 (+ 170us) server_negotiation.cc:299] Negotiation successful
0618 22:21:47.416368 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260618 22:21:47.416814 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821065380026-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821065380026-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821065380026-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:40847
--builtin_ntp_servers=127.0.98.84:44135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260618 22:21:47.417351 18176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:47.409322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:40847 (local address 127.0.98.65:43423)
0618 22:21:47.410048 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:47.410089 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0618 22:21:47.411340 (+ 1251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:21:47.411877 (+ 537us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:21:47.411884 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:21:47.411969 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:21:47.412428 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:47.412439 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:47.413229 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:47.413239 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:47.414222 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:47.414251 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:47.414833 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:47.414838 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:47.415053 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:47.415756 (+ 703us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 22:21:47.415758 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:21:47.415798 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 22:21:47.415858 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:21:47.416160 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:21:47.416171 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:21:47.416183 (+ 12us) client_negotiation.cc:769] Sending connection context
0618 22:21:47.416284 (+ 101us) client_negotiation.cc:240] Negotiation successful
0618 22:21:47.416496 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":111,"threads_started":1}
I20260618 22:21:47.418720 18174 heartbeater.cc:347] Connected to a master server at 127.0.98.126:40847
I20260618 22:21:47.418944 18174 heartbeater.cc:464] Registering TS with master...
I20260618 22:21:47.419452 18174 heartbeater.cc:511] Master 127.0.98.126:40847 requested a full tablet report, sending...
I20260618 22:21:47.420936 17986 ts_manager.cc:194] Registered new tserver with Master: d00f8d610fa74810981fc2c8c0ad2c7f (127.0.98.65:38215)
I20260618 22:21:47.422277 17986 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.65:43423
W20260618 22:21:47.525216 18177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:21:47.525488 18177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:21:47.525563 18177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:21:47.529500 18177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:21:47.529572 18177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:21:47.529659 18177 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260618 22:21:47.534272 18177 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:44135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821065380026-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821065380026-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:40847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18177
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821065380026-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:21:47.535187 18177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:21:47.536407 18177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:21:47.545725 18182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:21:47.545696 18185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:21:47.545807 18183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:21:47.545986 18177 server_base.cc:1034] running on GCE node
I20260618 22:21:47.546586 18177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:21:47.547214 18177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:21:47.548394 18177 hybrid_clock.cc:648] HybridClock initialized: now 1781821307548379 us; error 41 us; skew 500 ppm
I20260618 22:21:47.550165 18177 webserver.cc:460] Webserver started at http://127.0.98.66:43457/ using document root <none> and password file <none>
I20260618 22:21:47.550709 18177 fs_manager.cc:359] Metadata directory not provided
I20260618 22:21:47.550798 18177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:21:47.551017 18177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:21:47.552964 18177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821065380026-393-0/minicluster-data/ts-1/data/instance:
uuid: "7369219284d44fc79da1fdc8f6438db3"
format_stamp: "Formatted at 2026-06-18 22:21:47 on dist-test-slave-g6r4"
server_key: "4a4bddf9bac886ed8e4c4add498f0c14"
server_key_iv: "1078c49299d97fed0a87f234effd5c55"
server_key_version: "encryptionkey@0"
I20260618 22:21:47.553543 18177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821065380026-393-0/minicluster-data/ts-1/wal/instance:
uuid: "7369219284d44fc79da1fdc8f6438db3"
format_stamp: "Formatted at 2026-06-18 22:21:47 on dist-test-slave-g6r4"
server_key: "4a4bddf9bac886ed8e4c4add498f0c14"
server_key_iv: "1078c49299d97fed0a87f234effd5c55"
server_key_version: "encryptionkey@0"
I20260618 22:21:47.557209 18177 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.001s
I20260618 22:21:47.559599 18191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:47.560765 18177 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260618 22:21:47.560899 18177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821065380026-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821065380026-393-0/minicluster-data/ts-1/wal
uuid: "7369219284d44fc79da1fdc8f6438db3"
format_stamp: "Formatted at 2026-06-18 22:21:47 on dist-test-slave-g6r4"
server_key: "4a4bddf9bac886ed8e4c4add498f0c14"
server_key_iv: "1078c49299d97fed0a87f234effd5c55"
server_key_version: "encryptionkey@0"
I20260618 22:21:47.561008 18177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821065380026-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821065380026-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821065380026-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:21:47.580953 18177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:21:47.581547 18177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:21:47.581735 18177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:21:47.582418 18177 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:21:47.583317 18177 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:21:47.583396 18177 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:47.583446 18177 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:21:47.583500 18177 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:47.592392 18177 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:33243
I20260618 22:21:47.592418 18304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:33243 every 8 connection(s)
I20260618 22:21:47.593230 18177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
I20260618 22:21:47.593726 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 18177
I20260618 22:21:47.593878 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821065380026-393-0/minicluster-data/ts-1/wal/instance
I20260618 22:21:47.594216 393 external_mini_cluster.cc:1428] Setting key 6061f7d390e2acc7a46660f763a5263e
I20260618 22:21:47.596359 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821065380026-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821065380026-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821065380026-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:40847
--builtin_ntp_servers=127.0.98.84:44135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260618 22:21:47.604465 18025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:47.595099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:58099 (local address 127.0.98.126:40847)
0618 22:21:47.595217 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:47.595222 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:47.596526 (+ 1304us) server_negotiation.cc:407] Connection header received
0618 22:21:47.598512 (+ 1986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:47.598517 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:47.598586 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:47.598688 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:21:47.599858 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:47.600457 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:47.602369 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:47.602577 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:47.602662 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:47.603996 (+ 1334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:21:47.604000 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:21:47.604002 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:21:47.604014 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:21:47.604032 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:21:47.604040 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:21:47.604096 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:47.604302 (+ 206us) server_negotiation.cc:299] Negotiation successful
0618 22:21:47.604353 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260618 22:21:47.605762 18307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:47.595611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:40847 (local address 127.0.98.66:58099)
0618 22:21:47.596369 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:47.596427 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0618 22:21:47.598310 (+ 1883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:21:47.598887 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:21:47.598898 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:21:47.599009 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:21:47.599709 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:47.599729 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:47.600606 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:47.600612 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:47.602245 (+ 1633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:47.602259 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:47.602744 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:47.602750 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:47.602989 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:47.603677 (+ 688us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 22:21:47.603681 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:21:47.603743 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 22:21:47.603834 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:21:47.604127 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:21:47.604139 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:21:47.604148 (+ 9us) client_negotiation.cc:769] Sending connection context
0618 22:21:47.604306 (+ 158us) client_negotiation.cc:240] Negotiation successful
0618 22:21:47.604642 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":109,"threads_started":1}
I20260618 22:21:47.607039 18305 heartbeater.cc:347] Connected to a master server at 127.0.98.126:40847
I20260618 22:21:47.607300 18305 heartbeater.cc:464] Registering TS with master...
I20260618 22:21:47.607802 18305 heartbeater.cc:511] Master 127.0.98.126:40847 requested a full tablet report, sending...
I20260618 22:21:47.608824 17986 ts_manager.cc:194] Registered new tserver with Master: 7369219284d44fc79da1fdc8f6438db3 (127.0.98.66:33243)
I20260618 22:21:47.609411 17986 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.66:58099
W20260618 22:21:47.708060 18308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:21:47.708357 18308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:21:47.708398 18308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:21:47.712517 18308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:21:47.712563 18308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:21:47.712659 18308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260618 22:21:47.717301 18308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:44135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821065380026-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821065380026-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:40847
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18308
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821065380026-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:21:47.718326 18308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:21:47.719604 18308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:21:47.728729 18313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:21:47.728894 18308 server_base.cc:1034] running on GCE node
W20260618 22:21:47.728724 18314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:21:47.728727 18316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:21:47.729383 18308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:21:47.730098 18308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:21:47.731279 18308 hybrid_clock.cc:648] HybridClock initialized: now 1781821307731268 us; error 27 us; skew 500 ppm
I20260618 22:21:47.732991 18308 webserver.cc:460] Webserver started at http://127.0.98.67:34803/ using document root <none> and password file <none>
I20260618 22:21:47.733530 18308 fs_manager.cc:359] Metadata directory not provided
I20260618 22:21:47.733623 18308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:21:47.733848 18308 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:21:47.735827 18308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821065380026-393-0/minicluster-data/ts-2/data/instance:
uuid: "1cd57bd8adad4f28bb8190e91f4f6a31"
format_stamp: "Formatted at 2026-06-18 22:21:47 on dist-test-slave-g6r4"
server_key: "cf86f7dfc57ec5e85f865fd69449f46e"
server_key_iv: "2e3b700aa40bb2270b7bbd6844a76b57"
server_key_version: "encryptionkey@0"
I20260618 22:21:47.736387 18308 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821065380026-393-0/minicluster-data/ts-2/wal/instance:
uuid: "1cd57bd8adad4f28bb8190e91f4f6a31"
format_stamp: "Formatted at 2026-06-18 22:21:47 on dist-test-slave-g6r4"
server_key: "cf86f7dfc57ec5e85f865fd69449f46e"
server_key_iv: "2e3b700aa40bb2270b7bbd6844a76b57"
server_key_version: "encryptionkey@0"
I20260618 22:21:47.739899 18308 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260618 22:21:47.742272 18322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:47.743443 18308 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:21:47.743572 18308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821065380026-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821065380026-393-0/minicluster-data/ts-2/wal
uuid: "1cd57bd8adad4f28bb8190e91f4f6a31"
format_stamp: "Formatted at 2026-06-18 22:21:47 on dist-test-slave-g6r4"
server_key: "cf86f7dfc57ec5e85f865fd69449f46e"
server_key_iv: "2e3b700aa40bb2270b7bbd6844a76b57"
server_key_version: "encryptionkey@0"
I20260618 22:21:47.743680 18308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821065380026-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821065380026-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821065380026-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:21:47.755571 18308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:21:47.756150 18308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:21:47.756352 18308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:21:47.756876 18308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:21:47.757723 18308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:21:47.757792 18308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:47.757860 18308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:21:47.757944 18308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:47.766618 18308 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:35455
I20260618 22:21:47.766649 18435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:35455 every 8 connection(s)
I20260618 22:21:47.767431 18308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
I20260618 22:21:47.772589 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 18308
I20260618 22:21:47.772713 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821065380026-393-0/minicluster-data/ts-2/wal/instance
I20260618 22:21:47.772987 393 external_mini_cluster.cc:1428] Setting key e5acddf5ef54efc275ac75fcbe63de44
I20260618 22:21:47.776394 18025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:47.769158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:39691 (local address 127.0.98.126:40847)
0618 22:21:47.769294 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:47.769298 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:47.770350 (+ 1052us) server_negotiation.cc:407] Connection header received
0618 22:21:47.771716 (+ 1366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:47.771719 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:47.771773 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:47.771881 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:21:47.772737 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:47.773272 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:47.774621 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:47.774879 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:47.774985 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:47.775976 (+ 991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:21:47.775979 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:21:47.775981 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:21:47.775993 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:21:47.776011 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:21:47.776019 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:21:47.776078 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:47.776240 (+ 162us) server_negotiation.cc:299] Negotiation successful
0618 22:21:47.776290 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260618 22:21:47.777182 18438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:47.769519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:40847 (local address 127.0.98.67:39691)
0618 22:21:47.770209 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:47.770249 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0618 22:21:47.771492 (+ 1243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:21:47.772041 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:21:47.772047 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:21:47.772125 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:21:47.772589 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:47.772601 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:47.773409 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:47.773417 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:47.774501 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:47.774512 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:47.775032 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:47.775036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:47.775240 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:47.775752 (+ 512us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 22:21:47.775753 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:21:47.775790 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 22:21:47.775846 (+ 56us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:21:47.776091 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:21:47.776097 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:21:47.776100 (+ 3us) client_negotiation.cc:769] Sending connection context
0618 22:21:47.776212 (+ 112us) client_negotiation.cc:240] Negotiation successful
0618 22:21:47.776407 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":119,"threads_started":1}
I20260618 22:21:47.778391 18436 heartbeater.cc:347] Connected to a master server at 127.0.98.126:40847
I20260618 22:21:47.778611 18436 heartbeater.cc:464] Registering TS with master...
I20260618 22:21:47.779146 18436 heartbeater.cc:511] Master 127.0.98.126:40847 requested a full tablet report, sending...
I20260618 22:21:47.780192 17986 ts_manager.cc:194] Registered new tserver with Master: 1cd57bd8adad4f28bb8190e91f4f6a31 (127.0.98.67:35455)
I20260618 22:21:47.780745 17986 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.67:39691
I20260618 22:21:47.786754 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 22:21:47.792141 18025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:47.788663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54080 (local address 127.0.98.126:40847)
0618 22:21:47.788804 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:47.788809 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:47.789006 (+ 197us) server_negotiation.cc:407] Connection header received
0618 22:21:47.789088 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:47.789090 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:47.789128 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:47.789190 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:21:47.789402 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:47.789920 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:47.790746 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:47.790950 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:47.791023 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:47.791752 (+ 729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:21:47.791755 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:21:47.791757 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:21:47.791768 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:21:47.791784 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:21:47.791792 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:21:47.791848 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:47.791917 (+ 69us) server_negotiation.cc:299] Negotiation successful
0618 22:21:47.791964 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260618 22:21:47.794945 17986 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:54080:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 22:21:47.797830 17986 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:21:47.811306 18447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:47.805830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:38215 (local address 127.0.0.1:57440)
0618 22:21:47.806213 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:47.806269 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0618 22:21:47.806382 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:21:47.807475 (+ 1093us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:21:47.807481 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:21:47.807502 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:21:47.807634 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:47.807641 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:47.808754 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:47.808761 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:47.810143 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:47.810152 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:47.810927 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:47.810930 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:47.811012 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:47.811041 (+ 29us) client_negotiation.cc:769] Sending connection context
0618 22:21:47.811113 (+ 72us) client_negotiation.cc:240] Negotiation successful
0618 22:21:47.811192 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":235,"thread_start_us":120,"threads_started":1}
I20260618 22:21:47.811424 18448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:47.806024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57440 (local address 127.0.98.65:38215)
0618 22:21:47.806971 (+ 947us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:47.806987 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:47.807011 (+ 24us) server_negotiation.cc:407] Connection header received
0618 22:21:47.807091 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:47.807096 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:47.807312 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:47.807473 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:21:47.807773 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:47.808645 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:47.810255 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:47.810822 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:47.810930 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:47.811120 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:47.811168 (+ 48us) server_negotiation.cc:299] Negotiation successful
0618 22:21:47.811260 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":842,"thread_start_us":116,"threads_started":1}
I20260618 22:21:47.812255 18450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:47.806605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:35455 (local address 127.0.0.1:35146)
0618 22:21:47.807196 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:47.807207 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 22:21:47.807279 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:21:47.807724 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:21:47.807727 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:21:47.807746 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:21:47.807830 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:47.807838 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:47.809249 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:47.809252 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:47.810643 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:47.810653 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:47.811877 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:47.811881 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:47.811967 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:47.811989 (+ 22us) client_negotiation.cc:769] Sending connection context
0618 22:21:47.812066 (+ 77us) client_negotiation.cc:240] Negotiation successful
0618 22:21:47.812127 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"mutex_wait_us":38,"thread_start_us":104,"threads_started":1}
I20260618 22:21:47.812374 18452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:47.806737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35146 (local address 127.0.98.67:35455)
0618 22:21:47.807283 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:47.807296 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:47.807318 (+ 22us) server_negotiation.cc:407] Connection header received
0618 22:21:47.807390 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:47.807396 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:47.807565 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:47.807739 (+ 174us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:21:47.807925 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:47.809105 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:47.810785 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:47.811558 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:47.811661 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:47.811818 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:47.812116 (+ 298us) server_negotiation.cc:299] Negotiation successful
0618 22:21:47.812210 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":112,"threads_started":1}
I20260618 22:21:47.813553 18449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:47.806143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:33243 (local address 127.0.0.1:47868)
0618 22:21:47.807115 (+ 972us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:47.807129 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:21:47.807224 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:21:47.807917 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:21:47.807920 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:21:47.807936 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:21:47.808007 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:47.808013 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:47.809354 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:47.809357 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:47.810643 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:47.810653 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:47.813179 (+ 2526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:47.813184 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:47.813275 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:47.813297 (+ 22us) client_negotiation.cc:769] Sending connection context
0618 22:21:47.813376 (+ 79us) client_negotiation.cc:240] Negotiation successful
0618 22:21:47.813437 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":907,"mutex_wait_us":90,"thread_start_us":86,"threads_started":1}
I20260618 22:21:47.813720 18451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:47.806640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47868 (local address 127.0.98.66:33243)
0618 22:21:47.807068 (+ 428us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:47.807080 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:47.807322 (+ 242us) server_negotiation.cc:407] Connection header received
0618 22:21:47.807430 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:47.807437 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:47.807640 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:47.807781 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:21:47.808143 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:47.809243 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:47.811598 (+ 2355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:47.812797 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:47.812899 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:47.813089 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:47.813441 (+ 352us) server_negotiation.cc:299] Negotiation successful
0618 22:21:47.813538 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":119,"threads_started":1}
I20260618 22:21:47.814791 18108 tablet_service.cc:1467] Processing CreateTablet for tablet cf5d7432704e4f0e80f7907d645c25da (DEFAULT_TABLE table=test-table [id=3756d0efcb3b4c49bbc948ad78d44abb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:21:47.815014 18370 tablet_service.cc:1467] Processing CreateTablet for tablet cf5d7432704e4f0e80f7907d645c25da (DEFAULT_TABLE table=test-table [id=3756d0efcb3b4c49bbc948ad78d44abb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:21:47.815850 18370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf5d7432704e4f0e80f7907d645c25da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:21:47.815871 18108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf5d7432704e4f0e80f7907d645c25da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:21:47.816275 18239 tablet_service.cc:1467] Processing CreateTablet for tablet cf5d7432704e4f0e80f7907d645c25da (DEFAULT_TABLE table=test-table [id=3756d0efcb3b4c49bbc948ad78d44abb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:21:47.817143 18239 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf5d7432704e4f0e80f7907d645c25da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:21:47.822243 18453 tablet_bootstrap.cc:492] T cf5d7432704e4f0e80f7907d645c25da P 1cd57bd8adad4f28bb8190e91f4f6a31: Bootstrap starting.
I20260618 22:21:47.823232 18454 tablet_bootstrap.cc:492] T cf5d7432704e4f0e80f7907d645c25da P d00f8d610fa74810981fc2c8c0ad2c7f: Bootstrap starting.
I20260618 22:21:47.823745 18455 tablet_bootstrap.cc:492] T cf5d7432704e4f0e80f7907d645c25da P 7369219284d44fc79da1fdc8f6438db3: Bootstrap starting.
I20260618 22:21:47.824486 18453 tablet_bootstrap.cc:654] T cf5d7432704e4f0e80f7907d645c25da P 1cd57bd8adad4f28bb8190e91f4f6a31: Neither blocks nor log segments found. Creating new log.
I20260618 22:21:47.825129 18454 tablet_bootstrap.cc:654] T cf5d7432704e4f0e80f7907d645c25da P d00f8d610fa74810981fc2c8c0ad2c7f: Neither blocks nor log segments found. Creating new log.
I20260618 22:21:47.825343 18453 log.cc:824] T cf5d7432704e4f0e80f7907d645c25da P 1cd57bd8adad4f28bb8190e91f4f6a31: Log is configured to *not* fsync() on all Append() calls
I20260618 22:21:47.825812 18455 tablet_bootstrap.cc:654] T cf5d7432704e4f0e80f7907d645c25da P 7369219284d44fc79da1fdc8f6438db3: Neither blocks nor log segments found. Creating new log.
I20260618 22:21:47.825932 18454 log.cc:824] T cf5d7432704e4f0e80f7907d645c25da P d00f8d610fa74810981fc2c8c0ad2c7f: Log is configured to *not* fsync() on all Append() calls
I20260618 22:21:47.826640 18455 log.cc:824] T cf5d7432704e4f0e80f7907d645c25da P 7369219284d44fc79da1fdc8f6438db3: Log is configured to *not* fsync() on all Append() calls
I20260618 22:21:47.827322 18453 tablet_bootstrap.cc:492] T cf5d7432704e4f0e80f7907d645c25da P 1cd57bd8adad4f28bb8190e91f4f6a31: No bootstrap required, opened a new log
I20260618 22:21:47.827548 18453 ts_tablet_manager.cc:1397] T cf5d7432704e4f0e80f7907d645c25da P 1cd57bd8adad4f28bb8190e91f4f6a31: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260618 22:21:47.827741 18454 tablet_bootstrap.cc:492] T cf5d7432704e4f0e80f7907d645c25da P d00f8d610fa74810981fc2c8c0ad2c7f: No bootstrap required, opened a new log
I20260618 22:21:47.827942 18454 ts_tablet_manager.cc:1397] T cf5d7432704e4f0e80f7907d645c25da P d00f8d610fa74810981fc2c8c0ad2c7f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260618 22:21:47.828329 18455 tablet_bootstrap.cc:492] T cf5d7432704e4f0e80f7907d645c25da P 7369219284d44fc79da1fdc8f6438db3: No bootstrap required, opened a new log
I20260618 22:21:47.828505 18455 ts_tablet_manager.cc:1397] T cf5d7432704e4f0e80f7907d645c25da P 7369219284d44fc79da1fdc8f6438db3: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260618 22:21:47.831452 18455 raft_consensus.cc:348] T cf5d7432704e4f0e80f7907d645c25da P 7369219284d44fc79da1fdc8f6438db3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00f8d610fa74810981fc2c8c0ad2c7f" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 38215 } } peers { permanent_uuid: "1cd57bd8adad4f28bb8190e91f4f6a31" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 35455 } } peers { permanent_uuid: "7369219284d44fc79da1fdc8f6438db3" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 33243 } }
I20260618 22:21:47.831676 18455 raft_consensus.cc:374] T cf5d7432704e4f0e80f7907d645c25da P 7369219284d44fc79da1fdc8f6438db3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:21:47.831745 18455 raft_consensus.cc:729] T cf5d7432704e4f0e80f7907d645c25da P 7369219284d44fc79da1fdc8f6438db3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7369219284d44fc79da1fdc8f6438db3, State: Initialized, Role: FOLLOWER
I20260618 22:21:47.831882 18454 raft_consensus.cc:348] T cf5d7432704e4f0e80f7907d645c25da P d00f8d610fa74810981fc2c8c0ad2c7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00f8d610fa74810981fc2c8c0ad2c7f" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 38215 } } peers { permanent_uuid: "1cd57bd8adad4f28bb8190e91f4f6a31" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 35455 } } peers { permanent_uuid: "7369219284d44fc79da1fdc8f6438db3" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 33243 } }
I20260618 22:21:47.831882 18453 raft_consensus.cc:348] T cf5d7432704e4f0e80f7907d645c25da P 1cd57bd8adad4f28bb8190e91f4f6a31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00f8d610fa74810981fc2c8c0ad2c7f" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 38215 } } peers { permanent_uuid: "1cd57bd8adad4f28bb8190e91f4f6a31" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 35455 } } peers { permanent_uuid: "7369219284d44fc79da1fdc8f6438db3" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 33243 } }
I20260618 22:21:47.832124 18453 raft_consensus.cc:374] T cf5d7432704e4f0e80f7907d645c25da P 1cd57bd8adad4f28bb8190e91f4f6a31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:21:47.832124 18454 raft_consensus.cc:374] T cf5d7432704e4f0e80f7907d645c25da P d00f8d610fa74810981fc2c8c0ad2c7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:21:47.832181 18454 raft_consensus.cc:729] T cf5d7432704e4f0e80f7907d645c25da P d00f8d610fa74810981fc2c8c0ad2c7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d00f8d610fa74810981fc2c8c0ad2c7f, State: Initialized, Role: FOLLOWER
I20260618 22:21:47.832218 18453 raft_consensus.cc:729] T cf5d7432704e4f0e80f7907d645c25da P 1cd57bd8adad4f28bb8190e91f4f6a31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cd57bd8adad4f28bb8190e91f4f6a31, State: Initialized, Role: FOLLOWER
I20260618 22:21:47.832190 18455 consensus_queue.cc:260] T cf5d7432704e4f0e80f7907d645c25da P 7369219284d44fc79da1fdc8f6438db3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00f8d610fa74810981fc2c8c0ad2c7f" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 38215 } } peers { permanent_uuid: "1cd57bd8adad4f28bb8190e91f4f6a31" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 35455 } } peers { permanent_uuid: "7369219284d44fc79da1fdc8f6438db3" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 33243 } }
I20260618 22:21:47.832639 18454 consensus_queue.cc:260] T cf5d7432704e4f0e80f7907d645c25da P d00f8d610fa74810981fc2c8c0ad2c7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00f8d610fa74810981fc2c8c0ad2c7f" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 38215 } } peers { permanent_uuid: "1cd57bd8adad4f28bb8190e91f4f6a31" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 35455 } } peers { permanent_uuid: "7369219284d44fc79da1fdc8f6438db3" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 33243 } }
I20260618 22:21:47.832639 18453 consensus_queue.cc:260] T cf5d7432704e4f0e80f7907d645c25da P 1cd57bd8adad4f28bb8190e91f4f6a31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00f8d610fa74810981fc2c8c0ad2c7f" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 38215 } } peers { permanent_uuid: "1cd57bd8adad4f28bb8190e91f4f6a31" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 35455 } } peers { permanent_uuid: "7369219284d44fc79da1fdc8f6438db3" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 33243 } }
I20260618 22:21:47.832785 18305 heartbeater.cc:503] Master 127.0.98.126:40847 was elected leader, sending a full tablet report...
I20260618 22:21:47.833595 18454 ts_tablet_manager.cc:1428] T cf5d7432704e4f0e80f7907d645c25da P d00f8d610fa74810981fc2c8c0ad2c7f: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260618 22:21:47.833595 18453 ts_tablet_manager.cc:1428] T cf5d7432704e4f0e80f7907d645c25da P 1cd57bd8adad4f28bb8190e91f4f6a31: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260618 22:21:47.833927 18174 heartbeater.cc:503] Master 127.0.98.126:40847 was elected leader, sending a full tablet report...
I20260618 22:21:47.833935 18436 heartbeater.cc:503] Master 127.0.98.126:40847 was elected leader, sending a full tablet report...
I20260618 22:21:47.834439 18455 ts_tablet_manager.cc:1428] T cf5d7432704e4f0e80f7907d645c25da P 7369219284d44fc79da1fdc8f6438db3: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
W20260618 22:21:47.844094 18306 tablet.cc:2369] T cf5d7432704e4f0e80f7907d645c25da P 7369219284d44fc79da1fdc8f6438db3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260618 22:21:47.908252 18175 tablet.cc:2369] T cf5d7432704e4f0e80f7907d645c25da P d00f8d610fa74810981fc2c8c0ad2c7f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260618 22:21:48.018436 18437 tablet.cc:2369] T cf5d7432704e4f0e80f7907d645c25da P 1cd57bd8adad4f28bb8190e91f4f6a31: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:21:48.278720 18461 raft_consensus.cc:482] T cf5d7432704e4f0e80f7907d645c25da P d00f8d610fa74810981fc2c8c0ad2c7f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:21:48.278957 18461 raft_consensus.cc:504] T cf5d7432704e4f0e80f7907d645c25da P d00f8d610fa74810981fc2c8c0ad2c7f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00f8d610fa74810981fc2c8c0ad2c7f" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 38215 } } peers { permanent_uuid: "1cd57bd8adad4f28bb8190e91f4f6a31" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 35455 } } peers { permanent_uuid: "7369219284d44fc79da1fdc8f6438db3" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 33243 } }
I20260618 22:21:48.279989 18460 raft_consensus.cc:482] T cf5d7432704e4f0e80f7907d645c25da P 1cd57bd8adad4f28bb8190e91f4f6a31 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:21:48.280194 18460 raft_consensus.cc:504] T cf5d7432704e4f0e80f7907d645c25da P 1cd57bd8adad4f28bb8190e91f4f6a31 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00f8d610fa74810981fc2c8c0ad2c7f" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 38215 } } peers { permanent_uuid: "1cd57bd8adad4f28bb8190e91f4f6a31" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 35455 } } peers { permanent_uuid: "7369219284d44fc79da1fdc8f6438db3" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 33243 } }
I20260618 22:21:48.280283 18461 leader_election.cc:288] T cf5d7432704e4f0e80f7907d645c25da P d00f8d610fa74810981fc2c8c0ad2c7f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cd57bd8adad4f28bb8190e91f4f6a31 (127.0.98.67:35455), 7369219284d44fc79da1fdc8f6438db3 (127.0.98.66:33243)
I20260618 22:21:48.281605 18460 leader_election.cc:288] T cf5d7432704e4f0e80f7907d645c25da P 1cd57bd8adad4f28bb8190e91f4f6a31 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d00f8d610fa74810981fc2c8c0ad2c7f (127.0.98.65:38215), 7369219284d44fc79da1fdc8f6438db3 (127.0.98.66:33243)
I20260618 22:21:48.285373 18463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:48.280495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:35455 (local address 127.0.98.65:59897)
0618 22:21:48.280950 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:48.280965 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:21:48.281074 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:21:48.281360 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:21:48.281363 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:21:48.281381 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:21:48.281512 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:48.281519 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:48.282623 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:48.282626 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:48.283916 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:48.283925 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:48.284826 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:48.284830 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:48.284916 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:48.284936 (+ 20us) client_negotiation.cc:769] Sending connection context
0618 22:21:48.285097 (+ 161us) client_negotiation.cc:240] Negotiation successful
0618 22:21:48.285179 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":375,"mutex_wait_us":1,"thread_start_us":89,"threads_started":1}
I20260618 22:21:48.285699 18451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:48.280614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:56243 (local address 127.0.98.66:33243)
0618 22:21:48.280773 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:48.280780 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:48.281075 (+ 295us) server_negotiation.cc:407] Connection header received
0618 22:21:48.281333 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:48.281339 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:48.281388 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:48.281457 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:21:48.281921 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:48.282656 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:48.284482 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:48.285046 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:48.285165 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:48.285230 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:48.285523 (+ 293us) server_negotiation.cc:299] Negotiation successful
0618 22:21:48.285587 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260618 22:21:48.286069 18462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:48.280510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:33243 (local address 127.0.98.65:56243)
0618 22:21:48.280990 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:48.281005 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:21:48.281149 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:21:48.281604 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:21:48.281610 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:21:48.281643 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:21:48.281753 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:48.281760 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:48.282891 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:48.282894 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:48.284366 (+ 1472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:48.284373 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:48.285288 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:48.285292 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:48.285392 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:48.285412 (+ 20us) client_negotiation.cc:769] Sending connection context
0618 22:21:48.285815 (+ 403us) client_negotiation.cc:240] Negotiation successful
0618 22:21:48.285886 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"mutex_wait_us":255,"thread_start_us":70,"threads_started":1}
I20260618 22:21:48.286160 18452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:48.280885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:59897 (local address 127.0.98.67:35455)
0618 22:21:48.281000 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:48.281006 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:48.281023 (+ 17us) server_negotiation.cc:407] Connection header received
0618 22:21:48.281203 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:48.281206 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:48.281255 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:48.281348 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:21:48.281644 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:48.282474 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:48.284029 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:48.284589 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:48.284701 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:48.284757 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:48.285048 (+ 291us) server_negotiation.cc:299] Negotiation successful
0618 22:21:48.286019 (+ 971us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":883,"server-negotiator.queue_time_us":42}
I20260618 22:21:48.286623 18448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:48.281997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:60875 (local address 127.0.98.65:38215)
0618 22:21:48.282091 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:48.282111 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:48.282380 (+ 269us) server_negotiation.cc:407] Connection header received
0618 22:21:48.282536 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:48.282539 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:48.282574 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:48.282613 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:21:48.282988 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:48.283726 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:48.285284 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:48.285872 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:48.285981 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:48.286026 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:48.286435 (+ 409us) server_negotiation.cc:299] Negotiation successful
0618 22:21:48.286518 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":39}
I20260618 22:21:48.286638 18390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf5d7432704e4f0e80f7907d645c25da" candidate_uuid: "d00f8d610fa74810981fc2c8c0ad2c7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cd57bd8adad4f28bb8190e91f4f6a31" is_pre_election: true
I20260618 22:21:48.286839 18390 raft_consensus.cc:2454] T cf5d7432704e4f0e80f7907d645c25da P 1cd57bd8adad4f28bb8190e91f4f6a31 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d00f8d610fa74810981fc2c8c0ad2c7f in term 0.
I20260618 22:21:48.286834 18259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf5d7432704e4f0e80f7907d645c25da" candidate_uuid: "d00f8d610fa74810981fc2c8c0ad2c7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7369219284d44fc79da1fdc8f6438db3" is_pre_election: true
I20260618 22:21:48.287133 18259 raft_consensus.cc:2454] T cf5d7432704e4f0e80f7907d645c25da P 7369219284d44fc79da1fdc8f6438db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d00f8d610fa74810981fc2c8c0ad2c7f in term 0.
I20260618 22:21:48.287274 18464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:48.282121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:33243 (local address 127.0.98.67:37641)
0618 22:21:48.282394 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:48.282407 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:21:48.282525 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:21:48.283457 (+ 932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:21:48.283460 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:21:48.283475 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:21:48.283597 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:48.283603 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:48.284701 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:48.284704 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:48.285922 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:48.285932 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:48.286900 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:48.286905 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:48.286997 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:48.287019 (+ 22us) client_negotiation.cc:769] Sending connection context
0618 22:21:48.287069 (+ 50us) client_negotiation.cc:240] Negotiation successful
0618 22:21:48.287138 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217,"mutex_wait_us":4,"thread_start_us":56,"threads_started":1}
I20260618 22:21:48.287479 18064 leader_election.cc:302] T cf5d7432704e4f0e80f7907d645c25da P d00f8d610fa74810981fc2c8c0ad2c7f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cd57bd8adad4f28bb8190e91f4f6a31, d00f8d610fa74810981fc2c8c0ad2c7f; no voters:
I20260618 22:21:48.287609 18465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:48.281726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:38215 (local address 127.0.98.67:60875)
0618 22:21:48.282306 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:48.282320 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:21:48.282439 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:21:48.282742 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:21:48.282745 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:21:48.282765 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:21:48.282880 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:48.282886 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:48.283857 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:48.283860 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:48.285165 (+ 1305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:48.285173 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:48.286188 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:48.286192 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:48.286286 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:48.286312 (+ 26us) client_negotiation.cc:769] Sending connection context
0618 22:21:48.286740 (+ 428us) client_negotiation.cc:240] Negotiation successful
0618 22:21:48.286812 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"mutex_wait_us":92,"thread_start_us":66,"threads_started":1}
I20260618 22:21:48.287794 18466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:48.282927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:37641 (local address 127.0.98.66:33243)
0618 22:21:48.283247 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:48.283251 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:48.283264 (+ 13us) server_negotiation.cc:407] Connection header received
0618 22:21:48.283301 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:48.283304 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:48.283340 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:48.283736 (+ 396us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:21:48.283836 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:48.284565 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:48.286121 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:48.286723 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:48.287518 (+ 795us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:48.287568 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:48.287605 (+ 37us) server_negotiation.cc:299] Negotiation successful
0618 22:21:48.287677 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":93,"threads_started":1}
I20260618 22:21:48.287849 18461 raft_consensus.cc:2781] T cf5d7432704e4f0e80f7907d645c25da P d00f8d610fa74810981fc2c8c0ad2c7f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:21:48.287954 18461 raft_consensus.cc:482] T cf5d7432704e4f0e80f7907d645c25da P d00f8d610fa74810981fc2c8c0ad2c7f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:21:48.288013 18461 raft_consensus.cc:3037] T cf5d7432704e4f0e80f7907d645c25da P d00f8d610fa74810981fc2c8c0ad2c7f [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:21:48.288161 18259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf5d7432704e4f0e80f7907d645c25da" candidate_uuid: "1cd57bd8adad4f28bb8190e91f4f6a31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7369219284d44fc79da1fdc8f6438db3" is_pre_election: true
I20260618 22:21:48.288298 18259 raft_consensus.cc:2454] T cf5d7432704e4f0e80f7907d645c25da P 7369219284d44fc79da1fdc8f6438db3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cd57bd8adad4f28bb8190e91f4f6a31 in term 0.
I20260618 22:21:48.288626 18128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf5d7432704e4f0e80f7907d645c25da" candidate_uuid: "1cd57bd8adad4f28bb8190e91f4f6a31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d00f8d610fa74810981fc2c8c0ad2c7f" is_pre_election: true
I20260618 22:21:48.288882 18324 leader_election.cc:302] T cf5d7432704e4f0e80f7907d645c25da P 1cd57bd8adad4f28bb8190e91f4f6a31 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cd57bd8adad4f28bb8190e91f4f6a31, 7369219284d44fc79da1fdc8f6438db3; no voters:
I20260618 22:21:48.289131 18460 raft_consensus.cc:2781] T cf5d7432704e4f0e80f7907d645c25da P 1cd57bd8adad4f28bb8190e91f4f6a31 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:21:48.289204 18460 raft_consensus.cc:482] T cf5d7432704e4f0e80f7907d645c25da P 1cd57bd8adad4f28bb8190e91f4f6a31 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:21:48.289234 18460 raft_consensus.cc:3037] T cf5d7432704e4f0e80f7907d645c25da P 1cd57bd8adad4f28bb8190e91f4f6a31 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:21:48.289397 18461 raft_consensus.cc:504] T cf5d7432704e4f0e80f7907d645c25da P d00f8d610fa74810981fc2c8c0ad2c7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00f8d610fa74810981fc2c8c0ad2c7f" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 38215 } } peers { permanent_uuid: "1cd57bd8adad4f28bb8190e91f4f6a31" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 35455 } } peers { permanent_uuid: "7369219284d44fc79da1fdc8f6438db3" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 33243 } }
I20260618 22:21:48.289645 18128 raft_consensus.cc:2379] T cf5d7432704e4f0e80f7907d645c25da P d00f8d610fa74810981fc2c8c0ad2c7f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 1cd57bd8adad4f28bb8190e91f4f6a31 in current term 1: Already voted for candidate d00f8d610fa74810981fc2c8c0ad2c7f in this term.
I20260618 22:21:48.289788 18461 leader_election.cc:288] T cf5d7432704e4f0e80f7907d645c25da P d00f8d610fa74810981fc2c8c0ad2c7f [CANDIDATE]: Term 1 election: Requested vote from peers 1cd57bd8adad4f28bb8190e91f4f6a31 (127.0.98.67:35455), 7369219284d44fc79da1fdc8f6438db3 (127.0.98.66:33243)
I20260618 22:21:48.290176 18390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf5d7432704e4f0e80f7907d645c25da" candidate_uuid: "d00f8d610fa74810981fc2c8c0ad2c7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cd57bd8adad4f28bb8190e91f4f6a31"
I20260618 22:21:48.290211 18259 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf5d7432704e4f0e80f7907d645c25da" candidate_uuid: "d00f8d610fa74810981fc2c8c0ad2c7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7369219284d44fc79da1fdc8f6438db3"
I20260618 22:21:48.290313 18259 raft_consensus.cc:3037] T cf5d7432704e4f0e80f7907d645c25da P 7369219284d44fc79da1fdc8f6438db3 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:21:48.290544 18460 raft_consensus.cc:504] T cf5d7432704e4f0e80f7907d645c25da P 1cd57bd8adad4f28bb8190e91f4f6a31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00f8d610fa74810981fc2c8c0ad2c7f" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 38215 } } peers { permanent_uuid: "1cd57bd8adad4f28bb8190e91f4f6a31" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 35455 } } peers { permanent_uuid: "7369219284d44fc79da1fdc8f6438db3" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 33243 } }
I20260618 22:21:48.290807 18390 raft_consensus.cc:2379] T cf5d7432704e4f0e80f7907d645c25da P 1cd57bd8adad4f28bb8190e91f4f6a31 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d00f8d610fa74810981fc2c8c0ad2c7f in current term 1: Already voted for candidate 1cd57bd8adad4f28bb8190e91f4f6a31 in this term.
I20260618 22:21:48.290885 18460 leader_election.cc:288] T cf5d7432704e4f0e80f7907d645c25da P 1cd57bd8adad4f28bb8190e91f4f6a31 [CANDIDATE]: Term 1 election: Requested vote from peers d00f8d610fa74810981fc2c8c0ad2c7f (127.0.98.65:38215), 7369219284d44fc79da1fdc8f6438db3 (127.0.98.66:33243)
I20260618 22:21:48.291225 18128 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf5d7432704e4f0e80f7907d645c25da" candidate_uuid: "1cd57bd8adad4f28bb8190e91f4f6a31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d00f8d610fa74810981fc2c8c0ad2c7f"
I20260618 22:21:48.291354 18128 raft_consensus.cc:2379] T cf5d7432704e4f0e80f7907d645c25da P d00f8d610fa74810981fc2c8c0ad2c7f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 1cd57bd8adad4f28bb8190e91f4f6a31 in current term 1: Already voted for candidate d00f8d610fa74810981fc2c8c0ad2c7f in this term.
I20260618 22:21:48.291440 18258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf5d7432704e4f0e80f7907d645c25da" candidate_uuid: "1cd57bd8adad4f28bb8190e91f4f6a31" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7369219284d44fc79da1fdc8f6438db3"
I20260618 22:21:48.291523 18259 raft_consensus.cc:2454] T cf5d7432704e4f0e80f7907d645c25da P 7369219284d44fc79da1fdc8f6438db3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d00f8d610fa74810981fc2c8c0ad2c7f in term 1.
I20260618 22:21:48.291781 18062 leader_election.cc:302] T cf5d7432704e4f0e80f7907d645c25da P d00f8d610fa74810981fc2c8c0ad2c7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7369219284d44fc79da1fdc8f6438db3, d00f8d610fa74810981fc2c8c0ad2c7f; no voters: 1cd57bd8adad4f28bb8190e91f4f6a31
I20260618 22:21:48.291931 18461 raft_consensus.cc:2781] T cf5d7432704e4f0e80f7907d645c25da P d00f8d610fa74810981fc2c8c0ad2c7f [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:21:48.291956 18324 leader_election.cc:302] T cf5d7432704e4f0e80f7907d645c25da P 1cd57bd8adad4f28bb8190e91f4f6a31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1cd57bd8adad4f28bb8190e91f4f6a31; no voters: 7369219284d44fc79da1fdc8f6438db3, d00f8d610fa74810981fc2c8c0ad2c7f
I20260618 22:21:48.292136 18460 raft_consensus.cc:2726] T cf5d7432704e4f0e80f7907d645c25da P 1cd57bd8adad4f28bb8190e91f4f6a31 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260618 22:21:48.292146 18461 raft_consensus.cc:686] T cf5d7432704e4f0e80f7907d645c25da P d00f8d610fa74810981fc2c8c0ad2c7f [term 1 LEADER]: Becoming Leader. State: Replica: d00f8d610fa74810981fc2c8c0ad2c7f, State: Running, Role: LEADER
I20260618 22:21:48.292488 18461 consensus_queue.cc:237] T cf5d7432704e4f0e80f7907d645c25da P d00f8d610fa74810981fc2c8c0ad2c7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00f8d610fa74810981fc2c8c0ad2c7f" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 38215 } } peers { permanent_uuid: "1cd57bd8adad4f28bb8190e91f4f6a31" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 35455 } } peers { permanent_uuid: "7369219284d44fc79da1fdc8f6438db3" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 33243 } }
I20260618 22:21:48.295403 17986 catalog_manager.cc:5495] T cf5d7432704e4f0e80f7907d645c25da P d00f8d610fa74810981fc2c8c0ad2c7f reported cstate change: term changed from 0 to 1, leader changed from <none> to d00f8d610fa74810981fc2c8c0ad2c7f (127.0.98.65). New cstate: current_term: 1 leader_uuid: "d00f8d610fa74810981fc2c8c0ad2c7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d00f8d610fa74810981fc2c8c0ad2c7f" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 38215 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cd57bd8adad4f28bb8190e91f4f6a31" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 35455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7369219284d44fc79da1fdc8f6438db3" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 33243 } health_report { overall_health: UNKNOWN } } }
I20260618 22:21:48.367853 18448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:48.363968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57442 (local address 127.0.98.65:38215)
0618 22:21:48.364128 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:48.364133 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:48.364278 (+ 145us) server_negotiation.cc:407] Connection header received
0618 22:21:48.364483 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:48.364486 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:48.364542 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:48.364657 (+ 115us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:21:48.364906 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:48.365474 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:48.366520 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:48.366746 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:48.366828 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:48.367065 (+ 237us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:21:48.367482 (+ 417us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:21:48.367572 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:48.367625 (+ 53us) server_negotiation.cc:299] Negotiation successful
0618 22:21:48.367667 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260618 22:21:48.373226 18390 raft_consensus.cc:1261] T cf5d7432704e4f0e80f7907d645c25da P 1cd57bd8adad4f28bb8190e91f4f6a31 [term 1 FOLLOWER]: Refusing update from remote peer d00f8d610fa74810981fc2c8c0ad2c7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:21:48.373250 18258 raft_consensus.cc:1261] T cf5d7432704e4f0e80f7907d645c25da P 7369219284d44fc79da1fdc8f6438db3 [term 1 FOLLOWER]: Refusing update from remote peer d00f8d610fa74810981fc2c8c0ad2c7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:21:48.373764 18461 consensus_queue.cc:1035] T cf5d7432704e4f0e80f7907d645c25da P d00f8d610fa74810981fc2c8c0ad2c7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cd57bd8adad4f28bb8190e91f4f6a31" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 35455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:21:48.373976 18467 consensus_queue.cc:1035] T cf5d7432704e4f0e80f7907d645c25da P d00f8d610fa74810981fc2c8c0ad2c7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7369219284d44fc79da1fdc8f6438db3" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 33243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:21:48.382522 18473 mvcc.cc:205] Tried to move back new op lower bound from 7298340079090221056 to 7298340078768320512. Current Snapshot: MvccSnapshot[applied={T|T < 7298340079090221056}]
I20260618 22:21:48.385286 18475 mvcc.cc:205] Tried to move back new op lower bound from 7298340079090221056 to 7298340078768320512. Current Snapshot: MvccSnapshot[applied={T|T < 7298340079090221056}]
I20260618 22:21:48.389057 17984 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:54080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:21:48.389189 17984 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:54080:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:21:48.391389 17984 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dbcdede77e4d40a8aafa6e1c9c933f98: Sending DeleteTablet for 3 replicas of tablet cf5d7432704e4f0e80f7907d645c25da
I20260618 22:21:48.392127 18370 tablet_service.cc:1514] Processing DeleteTablet for tablet cf5d7432704e4f0e80f7907d645c25da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:21:48 UTC) from {username='slave'} at 127.0.0.1:35146
I20260618 22:21:48.392210 18108 tablet_service.cc:1514] Processing DeleteTablet for tablet cf5d7432704e4f0e80f7907d645c25da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:21:48 UTC) from {username='slave'} at 127.0.0.1:57440
I20260618 22:21:48.392565 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 18046
I20260618 22:21:48.392601 18239 tablet_service.cc:1514] Processing DeleteTablet for tablet cf5d7432704e4f0e80f7907d645c25da with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:21:48 UTC) from {username='slave'} at 127.0.0.1:47868
I20260618 22:21:48.393308 18483 tablet_replica.cc:330] T cf5d7432704e4f0e80f7907d645c25da P 1cd57bd8adad4f28bb8190e91f4f6a31: stopping tablet replica
I20260618 22:21:48.393643 18483 raft_consensus.cc:2229] T cf5d7432704e4f0e80f7907d645c25da P 1cd57bd8adad4f28bb8190e91f4f6a31 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:21:48.393989 18483 raft_consensus.cc:2258] T cf5d7432704e4f0e80f7907d645c25da P 1cd57bd8adad4f28bb8190e91f4f6a31 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:21:48.394119 18484 tablet_replica.cc:330] T cf5d7432704e4f0e80f7907d645c25da P 7369219284d44fc79da1fdc8f6438db3: stopping tablet replica
I20260618 22:21:48.394430 18484 raft_consensus.cc:2229] T cf5d7432704e4f0e80f7907d645c25da P 7369219284d44fc79da1fdc8f6438db3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:21:48.394703 18484 raft_consensus.cc:2258] T cf5d7432704e4f0e80f7907d645c25da P 7369219284d44fc79da1fdc8f6438db3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:21:48.395303 18483 ts_tablet_manager.cc:1905] T cf5d7432704e4f0e80f7907d645c25da P 1cd57bd8adad4f28bb8190e91f4f6a31: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:21:48.395864 18484 ts_tablet_manager.cc:1905] T cf5d7432704e4f0e80f7907d645c25da P 7369219284d44fc79da1fdc8f6438db3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:21:48.398092 18483 ts_tablet_manager.cc:1918] T cf5d7432704e4f0e80f7907d645c25da P 1cd57bd8adad4f28bb8190e91f4f6a31: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:21:48.398190 18483 log.cc:1196] T cf5d7432704e4f0e80f7907d645c25da P 1cd57bd8adad4f28bb8190e91f4f6a31: Deleting WAL directory at /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821065380026-393-0/minicluster-data/ts-2/wal/wals/cf5d7432704e4f0e80f7907d645c25da
I20260618 22:21:48.398561 18483 ts_tablet_manager.cc:1939] T cf5d7432704e4f0e80f7907d645c25da P 1cd57bd8adad4f28bb8190e91f4f6a31: Deleting consensus metadata
I20260618 22:21:48.398562 18484 ts_tablet_manager.cc:1918] T cf5d7432704e4f0e80f7907d645c25da P 7369219284d44fc79da1fdc8f6438db3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:21:48.398638 18484 log.cc:1196] T cf5d7432704e4f0e80f7907d645c25da P 7369219284d44fc79da1fdc8f6438db3: Deleting WAL directory at /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781821065380026-393-0/minicluster-data/ts-1/wal/wals/cf5d7432704e4f0e80f7907d645c25da
I20260618 22:21:48.398959 18484 ts_tablet_manager.cc:1939] T cf5d7432704e4f0e80f7907d645c25da P 7369219284d44fc79da1fdc8f6438db3: Deleting consensus metadata
W20260618 22:21:48.399140 17973 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.65:38215 (error 108)
I20260618 22:21:48.399969 17973 catalog_manager.cc:4841] TS 1cd57bd8adad4f28bb8190e91f4f6a31 (127.0.98.67:35455): tablet cf5d7432704e4f0e80f7907d645c25da (table test-table [id=3756d0efcb3b4c49bbc948ad78d44abb]) successfully deleted
I20260618 22:21:48.400032 17971 catalog_manager.cc:4841] TS 7369219284d44fc79da1fdc8f6438db3 (127.0.98.66:33243): tablet cf5d7432704e4f0e80f7907d645c25da (table test-table [id=3756d0efcb3b4c49bbc948ad78d44abb]) successfully deleted
I20260618 22:21:48.400130 18486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:48.399760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:38215 (local address 127.0.0.1:57458)
0618 22:21:48.399953 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:48.400020 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.65:38215: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":75,"threads_started":1}
W20260618 22:21:48.400323 17973 catalog_manager.cc:4584] TS d00f8d610fa74810981fc2c8c0ad2c7f (127.0.98.65:38215): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cf5d7432704e4f0e80f7907d645c25da: Network error: Client connection negotiation failed: client connection to 127.0.98.65:38215: connect: Connection refused (error 111)
I20260618 22:21:48.400405 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 18177
I20260618 22:21:48.406181 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 18308
I20260618 22:21:48.411213 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 17955
2026-06-18T22:21:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1447 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:21:48 dist-test-slave-g6r4 krb5kdc[18489](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:21:48 dist-test-slave-g6r4 krb5kdc[18489](info): set up 2 sockets
Jun 18 22:21:48 dist-test-slave-g6r4 krb5kdc[18489](info): commencing operation
krb5kdc: starting...
W20260618 22:21:49.462153 393 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/krb5kdc/test-user.keytab.
Jun 18 22:21:49 dist-test-slave-g6r4 krb5kdc[18489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:21:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:21:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260618 22:21:49.578994 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:34319
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:36957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:34319
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:21:49.702812 18502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:21:49.703236 18502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:21:49.703307 18502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:21:49.708397 18502 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:21:49.708448 18502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:21:49.708467 18502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:21:49.708518 18502 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:21:49.708546 18502 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:21:49.716085 18502 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:36957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:34319
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:34319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18502
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:21:49.717080 18502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:21:49.718142 18502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260618 22:21:49.726374 18502 server_base.cc:1034] running on GCE node
W20260618 22:21:49.726330 18508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:21:49.726317 18507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:21:49.726307 18510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:21:49.727149 18502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:21:49.728281 18502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:21:49.729480 18502 hybrid_clock.cc:648] HybridClock initialized: now 1781821309729458 us; error 33 us; skew 500 ppm
Jun 18 22:21:49 dist-test-slave-g6r4 krb5kdc[18489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821309, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:21:49.732302 18502 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260618 22:21:49.733187 18502 webserver.cc:460] Webserver started at http://127.0.98.126:32839/ using document root <none> and password file <none>
I20260618 22:21:49.733759 18502 fs_manager.cc:359] Metadata directory not provided
I20260618 22:21:49.733852 18502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:21:49.734118 18502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:21:49.736192 18502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/minicluster-data/master-0/data/instance:
uuid: "555a52e41592429384d42cc1fe9ebcbc"
format_stamp: "Formatted at 2026-06-18 22:21:49 on dist-test-slave-g6r4"
server_key: "22689a58254a6b867b7356b8db1a6de9"
server_key_iv: "16797d607f46b1028f75b8dccf79cd15"
server_key_version: "encryptionkey@0"
I20260618 22:21:49.736783 18502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/minicluster-data/master-0/wal/instance:
uuid: "555a52e41592429384d42cc1fe9ebcbc"
format_stamp: "Formatted at 2026-06-18 22:21:49 on dist-test-slave-g6r4"
server_key: "22689a58254a6b867b7356b8db1a6de9"
server_key_iv: "16797d607f46b1028f75b8dccf79cd15"
server_key_version: "encryptionkey@0"
I20260618 22:21:49.740557 18502 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260618 22:21:49.743000 18517 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:49.744266 18502 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260618 22:21:49.744422 18502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/minicluster-data/master-0/wal
uuid: "555a52e41592429384d42cc1fe9ebcbc"
format_stamp: "Formatted at 2026-06-18 22:21:49 on dist-test-slave-g6r4"
server_key: "22689a58254a6b867b7356b8db1a6de9"
server_key_iv: "16797d607f46b1028f75b8dccf79cd15"
server_key_version: "encryptionkey@0"
I20260618 22:21:49.744534 18502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:21:49.765019 18502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:21:49.768180 18502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:21:49.768379 18502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:21:49.775528 18502 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:34319
I20260618 22:21:49.775539 18569 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:34319 every 8 connection(s)
I20260618 22:21:49.776450 18502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
I20260618 22:21:49.779202 18570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:21:49.784428 18570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 555a52e41592429384d42cc1fe9ebcbc: Bootstrap starting.
I20260618 22:21:49.784650 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 18502
I20260618 22:21:49.784799 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/minicluster-data/master-0/wal/instance
I20260618 22:21:49.785096 393 external_mini_cluster.cc:1428] Setting key 0842b0720f6041ac51597c92f13047c3
I20260618 22:21:49.787099 18570 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 555a52e41592429384d42cc1fe9ebcbc: Neither blocks nor log segments found. Creating new log.
I20260618 22:21:49.787812 18570 log.cc:824] T 00000000000000000000000000000000 P 555a52e41592429384d42cc1fe9ebcbc: Log is configured to *not* fsync() on all Append() calls
I20260618 22:21:49.789517 18570 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 555a52e41592429384d42cc1fe9ebcbc: No bootstrap required, opened a new log
I20260618 22:21:49.793040 18570 raft_consensus.cc:348] T 00000000000000000000000000000000 P 555a52e41592429384d42cc1fe9ebcbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "555a52e41592429384d42cc1fe9ebcbc" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34319 } }
I20260618 22:21:49.793342 18570 raft_consensus.cc:374] T 00000000000000000000000000000000 P 555a52e41592429384d42cc1fe9ebcbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:21:49.793401 18570 raft_consensus.cc:729] T 00000000000000000000000000000000 P 555a52e41592429384d42cc1fe9ebcbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 555a52e41592429384d42cc1fe9ebcbc, State: Initialized, Role: FOLLOWER
Jun 18 22:21:49 dist-test-slave-g6r4 krb5kdc[18489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:21:49.793922 18570 consensus_queue.cc:260] T 00000000000000000000000000000000 P 555a52e41592429384d42cc1fe9ebcbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "555a52e41592429384d42cc1fe9ebcbc" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34319 } }
I20260618 22:21:49.794101 18570 raft_consensus.cc:388] T 00000000000000000000000000000000 P 555a52e41592429384d42cc1fe9ebcbc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:21:49.794176 18570 raft_consensus.cc:482] T 00000000000000000000000000000000 P 555a52e41592429384d42cc1fe9ebcbc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:21:49.794281 18570 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 555a52e41592429384d42cc1fe9ebcbc [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:21:49.795388 18570 raft_consensus.cc:504] T 00000000000000000000000000000000 P 555a52e41592429384d42cc1fe9ebcbc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "555a52e41592429384d42cc1fe9ebcbc" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34319 } }
I20260618 22:21:49.795750 18570 leader_election.cc:302] T 00000000000000000000000000000000 P 555a52e41592429384d42cc1fe9ebcbc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 555a52e41592429384d42cc1fe9ebcbc; no voters:
I20260618 22:21:49.796056 18570 leader_election.cc:288] T 00000000000000000000000000000000 P 555a52e41592429384d42cc1fe9ebcbc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:21:49.796173 18575 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 555a52e41592429384d42cc1fe9ebcbc [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:21:49.796387 18575 raft_consensus.cc:686] T 00000000000000000000000000000000 P 555a52e41592429384d42cc1fe9ebcbc [term 1 LEADER]: Becoming Leader. State: Replica: 555a52e41592429384d42cc1fe9ebcbc, State: Running, Role: LEADER
I20260618 22:21:49.796742 18575 consensus_queue.cc:237] T 00000000000000000000000000000000 P 555a52e41592429384d42cc1fe9ebcbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "555a52e41592429384d42cc1fe9ebcbc" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34319 } }
I20260618 22:21:49.797397 18570 sys_catalog.cc:564] T 00000000000000000000000000000000 P 555a52e41592429384d42cc1fe9ebcbc [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:21:49.798367 18576 sys_catalog.cc:455] T 00000000000000000000000000000000 P 555a52e41592429384d42cc1fe9ebcbc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "555a52e41592429384d42cc1fe9ebcbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "555a52e41592429384d42cc1fe9ebcbc" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34319 } } }
I20260618 22:21:49.798498 18576 sys_catalog.cc:458] T 00000000000000000000000000000000 P 555a52e41592429384d42cc1fe9ebcbc [sys.catalog]: This master's current role is: LEADER
I20260618 22:21:49.798890 18577 sys_catalog.cc:455] T 00000000000000000000000000000000 P 555a52e41592429384d42cc1fe9ebcbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 555a52e41592429384d42cc1fe9ebcbc. Latest consensus state: current_term: 1 leader_uuid: "555a52e41592429384d42cc1fe9ebcbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "555a52e41592429384d42cc1fe9ebcbc" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34319 } } }
I20260618 22:21:49.799503 18577 sys_catalog.cc:458] T 00000000000000000000000000000000 P 555a52e41592429384d42cc1fe9ebcbc [sys.catalog]: This master's current role is: LEADER
I20260618 22:21:49.800275 18573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:49.786402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58580 (local address 127.0.98.126:34319)
0618 22:21:49.786816 (+ 414us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:49.786830 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:49.786863 (+ 33us) server_negotiation.cc:407] Connection header received
0618 22:21:49.787529 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:49.787547 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:49.787855 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:49.788218 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:21:49.789003 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:49.790098 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:49.790937 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:49.791404 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:49.791591 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:49.794428 (+ 2837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:21:49.794434 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:21:49.794456 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:21:49.794481 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:21:49.796427 (+ 1946us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:21:49.796958 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:21:49.796964 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:21:49.796969 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:21:49.797104 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:21:49.797431 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:21:49.797435 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:21:49.797437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:21:49.797975 (+ 538us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:21:49.798178 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:49.799080 (+ 902us) server_negotiation.cc:299] Negotiation successful
0618 22:21:49.799374 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":123,"threads_started":1}
W20260618 22:21:49.802901 18591 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 555a52e41592429384d42cc1fe9ebcbc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 22:21:49.802964 18591 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 22:21:49.803045 18584 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:21:49.803733 18584 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:21:49.808599 18584 catalog_manager.cc:1347] Generated new cluster ID: d403646fc1aa49be9ce4d46383917665
I20260618 22:21:49.808677 18584 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:21:49.831125 18584 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:21:49.832337 18584 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:21:49.838832 18584 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 555a52e41592429384d42cc1fe9ebcbc: Generated new TSK 0
I20260618 22:21:49.839572 18584 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab.
I20260618 22:21:49.868830 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:34319
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:36957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:21:49.976317 18596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:21:49.976553 18596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:21:49.976593 18596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:21:49.980456 18596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:21:49.980508 18596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:21:49.980562 18596 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260618 22:21:49.985579 18596 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:36957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:34319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:21:49.986507 18596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:21:49.987411 18596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:21:49.996502 18602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:21:49.996600 18596 server_base.cc:1034] running on GCE node
W20260618 22:21:49.996533 18601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:21:49.996778 18604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:21:49.997134 18596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:21:49.997821 18596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:21:49.999007 18596 hybrid_clock.cc:648] HybridClock initialized: now 1781821309998990 us; error 29 us; skew 500 ppm
Jun 18 22:21:49 dist-test-slave-g6r4 krb5kdc[18489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821309, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:21:50.001681 18596 init.cc:377] Logged in from keytab as kudu/127.0.98.65@KRBTEST.COM (short username kudu)
I20260618 22:21:50.002554 18596 webserver.cc:460] Webserver started at http://127.0.98.65:35039/ using document root <none> and password file <none>
I20260618 22:21:50.003098 18596 fs_manager.cc:359] Metadata directory not provided
I20260618 22:21:50.003204 18596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:21:50.003427 18596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:21:50.005417 18596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/minicluster-data/ts-0/data/instance:
uuid: "cea947528dbd428990dc14e73f624272"
format_stamp: "Formatted at 2026-06-18 22:21:50 on dist-test-slave-g6r4"
server_key: "0a77f3a244b6d5328129ae77c551d7ca"
server_key_iv: "bb107783296398d5b706e29f2a348734"
server_key_version: "encryptionkey@0"
I20260618 22:21:50.006035 18596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/minicluster-data/ts-0/wal/instance:
uuid: "cea947528dbd428990dc14e73f624272"
format_stamp: "Formatted at 2026-06-18 22:21:50 on dist-test-slave-g6r4"
server_key: "0a77f3a244b6d5328129ae77c551d7ca"
server_key_iv: "bb107783296398d5b706e29f2a348734"
server_key_version: "encryptionkey@0"
I20260618 22:21:50.009617 18596 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260618 22:21:50.011966 18611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:50.013051 18596 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260618 22:21:50.013190 18596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/minicluster-data/ts-0/wal
uuid: "cea947528dbd428990dc14e73f624272"
format_stamp: "Formatted at 2026-06-18 22:21:50 on dist-test-slave-g6r4"
server_key: "0a77f3a244b6d5328129ae77c551d7ca"
server_key_iv: "bb107783296398d5b706e29f2a348734"
server_key_version: "encryptionkey@0"
I20260618 22:21:50.013302 18596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:21:50.029637 18596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:21:50.032313 18596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:21:50.032532 18596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:21:50.033100 18596 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:21:50.033875 18596 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:21:50.033975 18596 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:50.034045 18596 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:21:50.034101 18596 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:50.042976 18596 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:33733
I20260618 22:21:50.043007 18724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:33733 every 8 connection(s)
I20260618 22:21:50.043810 18596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
I20260618 22:21:50.044780 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 18596
I20260618 22:21:50.044914 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/minicluster-data/ts-0/wal/instance
I20260618 22:21:50.045234 393 external_mini_cluster.cc:1428] Setting key 205dd9886e9cff18ab03845def7bfde0
Jun 18 22:21:50 dist-test-slave-g6r4 krb5kdc[18489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821309, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.66@KRBTEST.COM" created.
I20260618 22:21:50.058758 18573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:50.045698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:39353 (local address 127.0.98.126:34319)
0618 22:21:50.045851 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:50.045855 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:50.046736 (+ 881us) server_negotiation.cc:407] Connection header received
0618 22:21:50.048191 (+ 1455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:50.048199 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:50.048241 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:50.048308 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:21:50.049844 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:50.050402 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:50.052229 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:50.052450 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:50.052527 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:50.055197 (+ 2670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:21:50.055204 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:21:50.055208 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:21:50.055242 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:21:50.056561 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:21:50.057114 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:21:50.057116 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:21:50.057117 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:21:50.057188 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:21:50.057565 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:21:50.057571 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:21:50.057576 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:21:50.057796 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:21:50.057874 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:50.058535 (+ 661us) server_negotiation.cc:299] Negotiation successful
0618 22:21:50.058643 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260618 22:21:50.059628 18727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:50.046103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34319 (local address 127.0.98.65:39353)
0618 22:21:50.046601 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:50.046658 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0618 22:21:50.047931 (+ 1273us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:21:50.048487 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:21:50.048496 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:21:50.048976 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:21:50.049698 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:50.049721 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:50.050535 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:50.050542 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:50.052105 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:50.052121 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:50.052596 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:50.052602 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:50.052806 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:50.053386 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:21:50.053412 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:21:50.055000 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:21:50.056667 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:21:50.056671 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:21:50.056681 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:21:50.057013 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:21:50.057297 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:21:50.057303 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:21:50.057307 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:21:50.057448 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:21:50.057932 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:21:50.057941 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:21:50.058269 (+ 328us) client_negotiation.cc:769] Sending connection context
0618 22:21:50.058478 (+ 209us) client_negotiation.cc:240] Negotiation successful
0618 22:21:50.058820 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":99,"threads_started":1}
I20260618 22:21:50.060784 18725 heartbeater.cc:347] Connected to a master server at 127.0.98.126:34319
I20260618 22:21:50.061031 18725 heartbeater.cc:464] Registering TS with master...
I20260618 22:21:50.061633 18725 heartbeater.cc:511] Master 127.0.98.126:34319 requested a full tablet report, sending...
I20260618 22:21:50.063205 18534 ts_manager.cc:194] Registered new tserver with Master: cea947528dbd428990dc14e73f624272 (127.0.98.65:33733)
I20260618 22:21:50.064306 18534 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.65@KRBTEST.COM'} at 127.0.98.65:39353
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab.
I20260618 22:21:50.073051 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:34319
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:36957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:21:50.177510 18730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:21:50.177745 18730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:21:50.177785 18730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:21:50.181612 18730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:21:50.181663 18730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:21:50.181717 18730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260618 22:21:50.186661 18730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:36957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:34319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:21:50.187557 18730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:21:50.188470 18730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:21:50.197505 18735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:21:50.197512 18738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:21:50.197548 18736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:21:50.197839 18730 server_base.cc:1034] running on GCE node
I20260618 22:21:50.198298 18730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:21:50.198920 18730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:21:50.200111 18730 hybrid_clock.cc:648] HybridClock initialized: now 1781821310200083 us; error 47 us; skew 500 ppm
Jun 18 22:21:50 dist-test-slave-g6r4 krb5kdc[18489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821310, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:21:50.202714 18730 init.cc:377] Logged in from keytab as kudu/127.0.98.66@KRBTEST.COM (short username kudu)
I20260618 22:21:50.203581 18730 webserver.cc:460] Webserver started at http://127.0.98.66:41981/ using document root <none> and password file <none>
I20260618 22:21:50.204124 18730 fs_manager.cc:359] Metadata directory not provided
I20260618 22:21:50.204213 18730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:21:50.204434 18730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:21:50.206406 18730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/minicluster-data/ts-1/data/instance:
uuid: "1f482de249fc4d21ad5da40eed04ef28"
format_stamp: "Formatted at 2026-06-18 22:21:50 on dist-test-slave-g6r4"
server_key: "105b1c57004af2ce2d7a8af1debb2b91"
server_key_iv: "39cdfdfb23044311f5afa1b35a1b7ae3"
server_key_version: "encryptionkey@0"
I20260618 22:21:50.206972 18730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/minicluster-data/ts-1/wal/instance:
uuid: "1f482de249fc4d21ad5da40eed04ef28"
format_stamp: "Formatted at 2026-06-18 22:21:50 on dist-test-slave-g6r4"
server_key: "105b1c57004af2ce2d7a8af1debb2b91"
server_key_iv: "39cdfdfb23044311f5afa1b35a1b7ae3"
server_key_version: "encryptionkey@0"
I20260618 22:21:50.210451 18730 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:21:50.212670 18745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:50.213745 18730 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:21:50.213874 18730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/minicluster-data/ts-1/wal
uuid: "1f482de249fc4d21ad5da40eed04ef28"
format_stamp: "Formatted at 2026-06-18 22:21:50 on dist-test-slave-g6r4"
server_key: "105b1c57004af2ce2d7a8af1debb2b91"
server_key_iv: "39cdfdfb23044311f5afa1b35a1b7ae3"
server_key_version: "encryptionkey@0"
I20260618 22:21:50.214028 18730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:21:50.229202 18730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:21:50.232501 18730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:21:50.232695 18730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:21:50.233234 18730 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:21:50.233994 18730 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:21:50.234037 18730 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:50.234102 18730 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:21:50.234140 18730 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:50.242144 18730 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:43769
I20260618 22:21:50.242192 18858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:43769 every 8 connection(s)
I20260618 22:21:50.242998 18730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
I20260618 22:21:50.248255 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 18730
I20260618 22:21:50.248400 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/minicluster-data/ts-1/wal/instance
I20260618 22:21:50.248683 393 external_mini_cluster.cc:1428] Setting key 3a71367d2a60d8e40750a0dbf49101bb
Jun 18 22:21:50 dist-test-slave-g6r4 krb5kdc[18489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821310, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:21:50.256832 18573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:50.244668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:43383 (local address 127.0.98.126:34319)
0618 22:21:50.244791 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:50.244794 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:50.245640 (+ 846us) server_negotiation.cc:407] Connection header received
0618 22:21:50.246856 (+ 1216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:50.246859 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:50.246903 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:50.246985 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:21:50.248031 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:50.248703 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:50.250230 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:50.250449 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:50.250587 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:50.252902 (+ 2315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:21:50.252909 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:21:50.252915 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:21:50.252947 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:21:50.254774 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:21:50.255285 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:21:50.255292 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:21:50.255296 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:21:50.255393 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:21:50.255723 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:21:50.255729 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:21:50.255733 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:21:50.255973 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:21:50.256077 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:50.256499 (+ 422us) server_negotiation.cc:299] Negotiation successful
0618 22:21:50.256638 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260618 22:21:50.257434 18861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:50.245052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34319 (local address 127.0.98.66:43383)
0618 22:21:50.245495 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:50.245534 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0618 22:21:50.246668 (+ 1134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:21:50.247132 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:21:50.247138 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:21:50.247452 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:21:50.247884 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:50.247896 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:50.248868 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:50.248887 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:50.250114 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:50.250124 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:50.250620 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:50.250624 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:50.250794 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:50.251361 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:21:50.251376 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:21:50.252746 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:21:50.254890 (+ 2144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:21:50.254894 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:21:50.254903 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:21:50.255183 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:21:50.255495 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:21:50.255497 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:21:50.255499 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:21:50.255622 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:21:50.256074 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:21:50.256079 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:21:50.256274 (+ 195us) client_negotiation.cc:769] Sending connection context
0618 22:21:50.256414 (+ 140us) client_negotiation.cc:240] Negotiation successful
0618 22:21:50.256631 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":123,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.67@KRBTEST.COM; defaulting to no policy
I20260618 22:21:50.258540 18859 heartbeater.cc:347] Connected to a master server at 127.0.98.126:34319
I20260618 22:21:50.258734 18859 heartbeater.cc:464] Registering TS with master...
I20260618 22:21:50.259119 18859 heartbeater.cc:511] Master 127.0.98.126:34319 requested a full tablet report, sending...
I20260618 22:21:50.260216 18534 ts_manager.cc:194] Registered new tserver with Master: 1f482de249fc4d21ad5da40eed04ef28 (127.0.98.66:43769)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.67@KRBTEST.COM" created.
I20260618 22:21:50.260857 18534 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.66@KRBTEST.COM'} at 127.0.98.66:43383
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab.
I20260618 22:21:50.274755 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:34319
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:36957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:21:50.381573 18864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:21:50.381819 18864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:21:50.381861 18864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:21:50.385814 18864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:21:50.385867 18864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:21:50.385939 18864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260618 22:21:50.391010 18864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:36957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:34319
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:21:50.391888 18864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:21:50.392808 18864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:21:50.402187 18870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:21:50.402187 18869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:21:50.402254 18864 server_base.cc:1034] running on GCE node
W20260618 22:21:50.402215 18872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:21:50.402818 18864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:21:50.403499 18864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:21:50.404691 18864 hybrid_clock.cc:648] HybridClock initialized: now 1781821310404667 us; error 35 us; skew 500 ppm
Jun 18 22:21:50 dist-test-slave-g6r4 krb5kdc[18489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821310, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:21:50.407429 18864 init.cc:377] Logged in from keytab as kudu/127.0.98.67@KRBTEST.COM (short username kudu)
I20260618 22:21:50.408313 18864 webserver.cc:460] Webserver started at http://127.0.98.67:41339/ using document root <none> and password file <none>
I20260618 22:21:50.408820 18864 fs_manager.cc:359] Metadata directory not provided
I20260618 22:21:50.408876 18864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:21:50.409031 18864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:21:50.411041 18864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/minicluster-data/ts-2/data/instance:
uuid: "bb648d7294dc41009b425c5289ad24e8"
format_stamp: "Formatted at 2026-06-18 22:21:50 on dist-test-slave-g6r4"
server_key: "3e427f884b2598c0d30150ed84b9c00e"
server_key_iv: "d6db1aaaca32676034a3b0487b3ed33e"
server_key_version: "encryptionkey@0"
I20260618 22:21:50.411543 18864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/minicluster-data/ts-2/wal/instance:
uuid: "bb648d7294dc41009b425c5289ad24e8"
format_stamp: "Formatted at 2026-06-18 22:21:50 on dist-test-slave-g6r4"
server_key: "3e427f884b2598c0d30150ed84b9c00e"
server_key_iv: "d6db1aaaca32676034a3b0487b3ed33e"
server_key_version: "encryptionkey@0"
I20260618 22:21:50.415040 18864 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260618 22:21:50.417443 18879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:50.418776 18864 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:21:50.418903 18864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/minicluster-data/ts-2/wal
uuid: "bb648d7294dc41009b425c5289ad24e8"
format_stamp: "Formatted at 2026-06-18 22:21:50 on dist-test-slave-g6r4"
server_key: "3e427f884b2598c0d30150ed84b9c00e"
server_key_iv: "d6db1aaaca32676034a3b0487b3ed33e"
server_key_version: "encryptionkey@0"
I20260618 22:21:50.419021 18864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:21:50.439597 18864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:21:50.442479 18864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:21:50.442688 18864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:21:50.443416 18864 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:21:50.444311 18864 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:21:50.444358 18864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:50.444396 18864 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:21:50.444412 18864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:50.453439 18864 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:44501
I20260618 22:21:50.453462 18992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:44501 every 8 connection(s)
I20260618 22:21:50.454352 18864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
I20260618 22:21:50.459861 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 18864
I20260618 22:21:50.459977 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/minicluster-data/ts-2/wal/instance
I20260618 22:21:50.460247 393 external_mini_cluster.cc:1428] Setting key 146855a2610fb2eaf92b7ac7ae93ea24
Jun 18 22:21:50 dist-test-slave-g6r4 krb5kdc[18489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821310, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:21:50.467355 18573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:50.456024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:41057 (local address 127.0.98.126:34319)
0618 22:21:50.456157 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:50.456161 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:50.456917 (+ 756us) server_negotiation.cc:407] Connection header received
0618 22:21:50.458204 (+ 1287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:50.458208 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:50.458266 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:50.458366 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:21:50.459461 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:50.460243 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:50.461626 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:50.461943 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:50.462046 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:50.464279 (+ 2233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:21:50.464282 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:21:50.464285 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:21:50.464306 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:21:50.465699 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:21:50.466172 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:21:50.466175 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:21:50.466176 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:21:50.466249 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:21:50.466518 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:21:50.466519 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:21:50.466520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:21:50.466677 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:21:50.466747 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:50.467161 (+ 414us) server_negotiation.cc:299] Negotiation successful
0618 22:21:50.467247 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260618 22:21:50.468080 18995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:50.456355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34319 (local address 127.0.98.67:41057)
0618 22:21:50.456774 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:50.456813 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0618 22:21:50.458002 (+ 1189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:21:50.458529 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:21:50.458536 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:21:50.458878 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:21:50.459320 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:50.459333 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:50.460388 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:50.460391 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:50.461479 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:50.461490 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:50.462068 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:50.462071 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:50.462284 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:50.462792 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:21:50.462807 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:21:50.464131 (+ 1324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:21:50.465790 (+ 1659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:21:50.465794 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:21:50.465803 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:21:50.466090 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:21:50.466326 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:21:50.466328 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:21:50.466329 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:21:50.466453 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:21:50.466756 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:21:50.466761 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:21:50.466957 (+ 196us) client_negotiation.cc:769] Sending connection context
0618 22:21:50.467101 (+ 144us) client_negotiation.cc:240] Negotiation successful
0618 22:21:50.467330 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":108,"threads_started":1}
I20260618 22:21:50.469139 18993 heartbeater.cc:347] Connected to a master server at 127.0.98.126:34319
I20260618 22:21:50.469326 18993 heartbeater.cc:464] Registering TS with master...
I20260618 22:21:50.469732 18993 heartbeater.cc:511] Master 127.0.98.126:34319 requested a full tablet report, sending...
I20260618 22:21:50.470674 18534 ts_manager.cc:194] Registered new tserver with Master: bb648d7294dc41009b425c5289ad24e8 (127.0.98.67:44501)
I20260618 22:21:50.471251 18534 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.67@KRBTEST.COM'} at 127.0.98.67:41057
I20260618 22:21:50.473867 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 22:21:50.483141 18573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:50.475747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58590 (local address 127.0.98.126:34319)
0618 22:21:50.475910 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:50.475915 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:50.476070 (+ 155us) server_negotiation.cc:407] Connection header received
0618 22:21:50.476219 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:50.476223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:50.476265 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:50.476346 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:21:50.477024 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:50.477514 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:50.478400 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:50.478621 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:50.478727 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:50.480144 (+ 1417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:21:50.480148 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:21:50.480152 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:21:50.480181 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:21:50.481337 (+ 1156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:21:50.481828 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:21:50.481832 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:21:50.481833 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:21:50.481974 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:21:50.482295 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:21:50.482298 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:21:50.482299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:21:50.482492 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:21:50.482645 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:50.482904 (+ 259us) server_negotiation.cc:299] Negotiation successful
0618 22:21:50.483003 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260618 22:21:50.486003 18534 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58590:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 22:21:50.488837 18534 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:21:50.501892 19004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:50.496540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:44501 (local address 127.0.0.1:59592)
0618 22:21:50.496860 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:50.496902 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0618 22:21:50.497015 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:21:50.497764 (+ 749us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:21:50.497770 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:21:50.497796 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:21:50.497952 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:50.497963 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:50.499396 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:50.499401 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:50.500372 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:50.500385 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:50.501332 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:50.501336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:50.501442 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:50.501492 (+ 50us) client_negotiation.cc:769] Sending connection context
0618 22:21:50.501633 (+ 141us) client_negotiation.cc:240] Negotiation successful
0618 22:21:50.501750 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":96,"threads_started":1}
I20260618 22:21:50.501933 19005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:50.496657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59592 (local address 127.0.98.67:44501)
0618 22:21:50.496993 (+ 336us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:50.497003 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:50.497022 (+ 19us) server_negotiation.cc:407] Connection header received
0618 22:21:50.497176 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:50.497182 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:50.497385 (+ 203us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:50.497577 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:21:50.498153 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:50.499284 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:50.500507 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:50.501209 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:50.501331 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:50.501477 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:50.501664 (+ 187us) server_negotiation.cc:299] Negotiation successful
0618 22:21:50.501750 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":167,"threads_started":1}
I20260618 22:21:50.503237 19006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:50.497610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:43769 (local address 127.0.0.1:49112)
0618 22:21:50.498510 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:50.498522 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 22:21:50.498637 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:21:50.499146 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:21:50.499149 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:21:50.499166 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:21:50.499252 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:50.499256 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:50.500505 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:50.500508 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:50.501749 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:50.501760 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:50.502832 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:50.502838 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:50.502924 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:50.502948 (+ 24us) client_negotiation.cc:769] Sending connection context
0618 22:21:50.503042 (+ 94us) client_negotiation.cc:240] Negotiation successful
0618 22:21:50.503107 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":843,"mutex_wait_us":69,"thread_start_us":94,"threads_started":1}
I20260618 22:21:50.503328 19009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:50.498088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49112 (local address 127.0.98.66:43769)
0618 22:21:50.498429 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:50.498440 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:50.498616 (+ 176us) server_negotiation.cc:407] Connection header received
0618 22:21:50.498784 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:50.498789 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:50.498978 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:50.499184 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:21:50.499340 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:50.500372 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:50.501889 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:50.502704 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:50.502816 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:50.502949 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:50.503070 (+ 121us) server_negotiation.cc:299] Negotiation successful
0618 22:21:50.503170 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250,"thread_start_us":166,"threads_started":1}
I20260618 22:21:50.503662 19008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:50.497291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:33733 (local address 127.0.0.1:57968)
0618 22:21:50.497882 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:50.497934 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0618 22:21:50.498058 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:21:50.498709 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:21:50.498715 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:21:50.498764 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:21:50.498857 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:50.498862 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:50.500366 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:50.500369 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:50.501749 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:50.501760 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:50.503160 (+ 1400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:50.503163 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:50.503264 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:50.503285 (+ 21us) client_negotiation.cc:769] Sending connection context
0618 22:21:50.503458 (+ 173us) client_negotiation.cc:240] Negotiation successful
0618 22:21:50.503526 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"mutex_wait_us":65,"thread_start_us":91,"threads_started":1}
I20260618 22:21:50.504129 19007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:50.497455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57968 (local address 127.0.98.65:33733)
0618 22:21:50.498304 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:50.498315 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:50.498336 (+ 21us) server_negotiation.cc:407] Connection header received
0618 22:21:50.498404 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:50.498410 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:50.498586 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:50.498735 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:21:50.498984 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:50.500220 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:50.501889 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:50.502877 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:50.502975 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:50.503100 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:50.503943 (+ 843us) server_negotiation.cc:299] Negotiation successful
0618 22:21:50.504017 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":755,"thread_start_us":116,"threads_started":1}
I20260618 22:21:50.505017 18927 tablet_service.cc:1467] Processing CreateTablet for tablet 515eaf1692704fb2b4545919cee00426 (DEFAULT_TABLE table=test-table [id=fe1bbbf5ca8845e6b2f94983465914a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:21:50.505872 18927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 515eaf1692704fb2b4545919cee00426. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:21:50.506016 18793 tablet_service.cc:1467] Processing CreateTablet for tablet 515eaf1692704fb2b4545919cee00426 (DEFAULT_TABLE table=test-table [id=fe1bbbf5ca8845e6b2f94983465914a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:21:50.506956 18793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 515eaf1692704fb2b4545919cee00426. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:21:50.506661 18659 tablet_service.cc:1467] Processing CreateTablet for tablet 515eaf1692704fb2b4545919cee00426 (DEFAULT_TABLE table=test-table [id=fe1bbbf5ca8845e6b2f94983465914a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:21:50.507539 18659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 515eaf1692704fb2b4545919cee00426. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:21:50.511871 19010 tablet_bootstrap.cc:492] T 515eaf1692704fb2b4545919cee00426 P bb648d7294dc41009b425c5289ad24e8: Bootstrap starting.
I20260618 22:21:50.513536 19011 tablet_bootstrap.cc:492] T 515eaf1692704fb2b4545919cee00426 P 1f482de249fc4d21ad5da40eed04ef28: Bootstrap starting.
I20260618 22:21:50.513830 19010 tablet_bootstrap.cc:654] T 515eaf1692704fb2b4545919cee00426 P bb648d7294dc41009b425c5289ad24e8: Neither blocks nor log segments found. Creating new log.
I20260618 22:21:50.514453 19012 tablet_bootstrap.cc:492] T 515eaf1692704fb2b4545919cee00426 P cea947528dbd428990dc14e73f624272: Bootstrap starting.
I20260618 22:21:50.515754 19010 log.cc:824] T 515eaf1692704fb2b4545919cee00426 P bb648d7294dc41009b425c5289ad24e8: Log is configured to *not* fsync() on all Append() calls
I20260618 22:21:50.516228 19011 tablet_bootstrap.cc:654] T 515eaf1692704fb2b4545919cee00426 P 1f482de249fc4d21ad5da40eed04ef28: Neither blocks nor log segments found. Creating new log.
I20260618 22:21:50.516296 19012 tablet_bootstrap.cc:654] T 515eaf1692704fb2b4545919cee00426 P cea947528dbd428990dc14e73f624272: Neither blocks nor log segments found. Creating new log.
I20260618 22:21:50.517055 19011 log.cc:824] T 515eaf1692704fb2b4545919cee00426 P 1f482de249fc4d21ad5da40eed04ef28: Log is configured to *not* fsync() on all Append() calls
I20260618 22:21:50.517560 19012 log.cc:824] T 515eaf1692704fb2b4545919cee00426 P cea947528dbd428990dc14e73f624272: Log is configured to *not* fsync() on all Append() calls
I20260618 22:21:50.517891 19010 tablet_bootstrap.cc:492] T 515eaf1692704fb2b4545919cee00426 P bb648d7294dc41009b425c5289ad24e8: No bootstrap required, opened a new log
I20260618 22:21:50.518249 19010 ts_tablet_manager.cc:1397] T 515eaf1692704fb2b4545919cee00426 P bb648d7294dc41009b425c5289ad24e8: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260618 22:21:50.519016 19011 tablet_bootstrap.cc:492] T 515eaf1692704fb2b4545919cee00426 P 1f482de249fc4d21ad5da40eed04ef28: No bootstrap required, opened a new log
I20260618 22:21:50.519193 19011 ts_tablet_manager.cc:1397] T 515eaf1692704fb2b4545919cee00426 P 1f482de249fc4d21ad5da40eed04ef28: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260618 22:21:50.519277 19012 tablet_bootstrap.cc:492] T 515eaf1692704fb2b4545919cee00426 P cea947528dbd428990dc14e73f624272: No bootstrap required, opened a new log
I20260618 22:21:50.519447 19012 ts_tablet_manager.cc:1397] T 515eaf1692704fb2b4545919cee00426 P cea947528dbd428990dc14e73f624272: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260618 22:21:50.522066 19010 raft_consensus.cc:348] T 515eaf1692704fb2b4545919cee00426 P bb648d7294dc41009b425c5289ad24e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb648d7294dc41009b425c5289ad24e8" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 44501 } } peers { permanent_uuid: "cea947528dbd428990dc14e73f624272" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33733 } } peers { permanent_uuid: "1f482de249fc4d21ad5da40eed04ef28" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43769 } }
I20260618 22:21:50.522373 19010 raft_consensus.cc:374] T 515eaf1692704fb2b4545919cee00426 P bb648d7294dc41009b425c5289ad24e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:21:50.522434 19010 raft_consensus.cc:729] T 515eaf1692704fb2b4545919cee00426 P bb648d7294dc41009b425c5289ad24e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb648d7294dc41009b425c5289ad24e8, State: Initialized, Role: FOLLOWER
I20260618 22:21:50.522857 19010 consensus_queue.cc:260] T 515eaf1692704fb2b4545919cee00426 P bb648d7294dc41009b425c5289ad24e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb648d7294dc41009b425c5289ad24e8" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 44501 } } peers { permanent_uuid: "cea947528dbd428990dc14e73f624272" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33733 } } peers { permanent_uuid: "1f482de249fc4d21ad5da40eed04ef28" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43769 } }
I20260618 22:21:50.522797 19012 raft_consensus.cc:348] T 515eaf1692704fb2b4545919cee00426 P cea947528dbd428990dc14e73f624272 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb648d7294dc41009b425c5289ad24e8" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 44501 } } peers { permanent_uuid: "cea947528dbd428990dc14e73f624272" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33733 } } peers { permanent_uuid: "1f482de249fc4d21ad5da40eed04ef28" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43769 } }
I20260618 22:21:50.522835 19011 raft_consensus.cc:348] T 515eaf1692704fb2b4545919cee00426 P 1f482de249fc4d21ad5da40eed04ef28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb648d7294dc41009b425c5289ad24e8" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 44501 } } peers { permanent_uuid: "cea947528dbd428990dc14e73f624272" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33733 } } peers { permanent_uuid: "1f482de249fc4d21ad5da40eed04ef28" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43769 } }
I20260618 22:21:50.523027 19012 raft_consensus.cc:374] T 515eaf1692704fb2b4545919cee00426 P cea947528dbd428990dc14e73f624272 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:21:50.523074 19012 raft_consensus.cc:729] T 515eaf1692704fb2b4545919cee00426 P cea947528dbd428990dc14e73f624272 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cea947528dbd428990dc14e73f624272, State: Initialized, Role: FOLLOWER
I20260618 22:21:50.523101 19011 raft_consensus.cc:374] T 515eaf1692704fb2b4545919cee00426 P 1f482de249fc4d21ad5da40eed04ef28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:21:50.523205 19011 raft_consensus.cc:729] T 515eaf1692704fb2b4545919cee00426 P 1f482de249fc4d21ad5da40eed04ef28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f482de249fc4d21ad5da40eed04ef28, State: Initialized, Role: FOLLOWER
I20260618 22:21:50.523561 19012 consensus_queue.cc:260] T 515eaf1692704fb2b4545919cee00426 P cea947528dbd428990dc14e73f624272 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb648d7294dc41009b425c5289ad24e8" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 44501 } } peers { permanent_uuid: "cea947528dbd428990dc14e73f624272" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33733 } } peers { permanent_uuid: "1f482de249fc4d21ad5da40eed04ef28" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43769 } }
I20260618 22:21:50.523664 18993 heartbeater.cc:503] Master 127.0.98.126:34319 was elected leader, sending a full tablet report...
I20260618 22:21:50.524137 19010 ts_tablet_manager.cc:1428] T 515eaf1692704fb2b4545919cee00426 P bb648d7294dc41009b425c5289ad24e8: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260618 22:21:50.524439 19012 ts_tablet_manager.cc:1428] T 515eaf1692704fb2b4545919cee00426 P cea947528dbd428990dc14e73f624272: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260618 22:21:50.524600 18725 heartbeater.cc:503] Master 127.0.98.126:34319 was elected leader, sending a full tablet report...
I20260618 22:21:50.525337 19011 consensus_queue.cc:260] T 515eaf1692704fb2b4545919cee00426 P 1f482de249fc4d21ad5da40eed04ef28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb648d7294dc41009b425c5289ad24e8" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 44501 } } peers { permanent_uuid: "cea947528dbd428990dc14e73f624272" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33733 } } peers { permanent_uuid: "1f482de249fc4d21ad5da40eed04ef28" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43769 } }
I20260618 22:21:50.525996 18859 heartbeater.cc:503] Master 127.0.98.126:34319 was elected leader, sending a full tablet report...
I20260618 22:21:50.526271 19011 ts_tablet_manager.cc:1428] T 515eaf1692704fb2b4545919cee00426 P 1f482de249fc4d21ad5da40eed04ef28: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260618 22:21:50.544945 18726 tablet.cc:2369] T 515eaf1692704fb2b4545919cee00426 P cea947528dbd428990dc14e73f624272: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260618 22:21:50.705360 18994 tablet.cc:2369] T 515eaf1692704fb2b4545919cee00426 P bb648d7294dc41009b425c5289ad24e8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260618 22:21:50.744067 18860 tablet.cc:2369] T 515eaf1692704fb2b4545919cee00426 P 1f482de249fc4d21ad5da40eed04ef28: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:21:50.830729 19018 raft_consensus.cc:482] T 515eaf1692704fb2b4545919cee00426 P 1f482de249fc4d21ad5da40eed04ef28 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:21:50.830997 19018 raft_consensus.cc:504] T 515eaf1692704fb2b4545919cee00426 P 1f482de249fc4d21ad5da40eed04ef28 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb648d7294dc41009b425c5289ad24e8" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 44501 } } peers { permanent_uuid: "cea947528dbd428990dc14e73f624272" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33733 } } peers { permanent_uuid: "1f482de249fc4d21ad5da40eed04ef28" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43769 } }
I20260618 22:21:50.832108 19018 leader_election.cc:288] T 515eaf1692704fb2b4545919cee00426 P 1f482de249fc4d21ad5da40eed04ef28 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bb648d7294dc41009b425c5289ad24e8 (127.0.98.67:44501), cea947528dbd428990dc14e73f624272 (127.0.98.65:33733)
I20260618 22:21:50.836263 19007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:50.832983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:53323 (local address 127.0.98.65:33733)
0618 22:21:50.833159 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:50.833164 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:50.833295 (+ 131us) server_negotiation.cc:407] Connection header received
0618 22:21:50.833368 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:50.833373 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:50.833431 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:50.833500 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:21:50.833775 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:50.834295 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:50.835288 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:50.835677 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:50.835786 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:50.835844 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:50.836061 (+ 217us) server_negotiation.cc:299] Negotiation successful
0618 22:21:50.836131 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260618 22:21:50.836272 19005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:50.832535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:60335 (local address 127.0.98.67:44501)
0618 22:21:50.832785 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:50.832792 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:50.832815 (+ 23us) server_negotiation.cc:407] Connection header received
0618 22:21:50.832948 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:50.832951 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:50.833006 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:50.833109 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:21:50.833453 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:50.834162 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:50.835287 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:50.835684 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:50.835801 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:50.835858 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:50.836101 (+ 243us) server_negotiation.cc:299] Negotiation successful
0618 22:21:50.836167 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154}
I20260618 22:21:50.836282 19020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:50.832824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:33733 (local address 127.0.98.66:53323)
0618 22:21:50.833187 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:50.833202 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:21:50.833296 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:21:50.833537 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:21:50.833539 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:21:50.833555 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:21:50.833668 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:50.833673 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:50.834399 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:50.834402 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:50.835163 (+ 761us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:50.835171 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:50.835795 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:50.835799 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:50.835894 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:50.835956 (+ 62us) client_negotiation.cc:769] Sending connection context
0618 22:21:50.836015 (+ 59us) client_negotiation.cc:240] Negotiation successful
0618 22:21:50.836099 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":22,"thread_start_us":77,"threads_started":1}
I20260618 22:21:50.836292 19019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:50.832434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:44501 (local address 127.0.98.66:60335)
0618 22:21:50.832689 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:50.832706 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0618 22:21:50.832812 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:21:50.833147 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:21:50.833150 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:21:50.833172 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:21:50.833329 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:50.833337 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:50.834270 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:50.834274 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:50.835162 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:50.835172 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:50.835817 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:50.835820 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:50.835962 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:50.835979 (+ 17us) client_negotiation.cc:769] Sending connection context
0618 22:21:50.836074 (+ 95us) client_negotiation.cc:240] Negotiation successful
0618 22:21:50.836146 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"mutex_wait_us":5,"thread_start_us":84,"threads_started":1}
I20260618 22:21:50.836968 18679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "515eaf1692704fb2b4545919cee00426" candidate_uuid: "1f482de249fc4d21ad5da40eed04ef28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea947528dbd428990dc14e73f624272" is_pre_election: true
I20260618 22:21:50.837294 18947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "515eaf1692704fb2b4545919cee00426" candidate_uuid: "1f482de249fc4d21ad5da40eed04ef28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb648d7294dc41009b425c5289ad24e8" is_pre_election: true
I20260618 22:21:50.837553 18947 raft_consensus.cc:2454] T 515eaf1692704fb2b4545919cee00426 P bb648d7294dc41009b425c5289ad24e8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f482de249fc4d21ad5da40eed04ef28 in term 0.
I20260618 22:21:50.837483 18679 raft_consensus.cc:2454] T 515eaf1692704fb2b4545919cee00426 P cea947528dbd428990dc14e73f624272 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1f482de249fc4d21ad5da40eed04ef28 in term 0.
I20260618 22:21:50.838115 18749 leader_election.cc:302] T 515eaf1692704fb2b4545919cee00426 P 1f482de249fc4d21ad5da40eed04ef28 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f482de249fc4d21ad5da40eed04ef28, bb648d7294dc41009b425c5289ad24e8; no voters:
I20260618 22:21:50.838349 19018 raft_consensus.cc:2781] T 515eaf1692704fb2b4545919cee00426 P 1f482de249fc4d21ad5da40eed04ef28 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:21:50.838423 19018 raft_consensus.cc:482] T 515eaf1692704fb2b4545919cee00426 P 1f482de249fc4d21ad5da40eed04ef28 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:21:50.838452 19018 raft_consensus.cc:3037] T 515eaf1692704fb2b4545919cee00426 P 1f482de249fc4d21ad5da40eed04ef28 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:21:50.839509 19018 raft_consensus.cc:504] T 515eaf1692704fb2b4545919cee00426 P 1f482de249fc4d21ad5da40eed04ef28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb648d7294dc41009b425c5289ad24e8" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 44501 } } peers { permanent_uuid: "cea947528dbd428990dc14e73f624272" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33733 } } peers { permanent_uuid: "1f482de249fc4d21ad5da40eed04ef28" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43769 } }
I20260618 22:21:50.839846 19018 leader_election.cc:288] T 515eaf1692704fb2b4545919cee00426 P 1f482de249fc4d21ad5da40eed04ef28 [CANDIDATE]: Term 1 election: Requested vote from peers bb648d7294dc41009b425c5289ad24e8 (127.0.98.67:44501), cea947528dbd428990dc14e73f624272 (127.0.98.65:33733)
I20260618 22:21:50.840333 18679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "515eaf1692704fb2b4545919cee00426" candidate_uuid: "1f482de249fc4d21ad5da40eed04ef28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cea947528dbd428990dc14e73f624272"
I20260618 22:21:50.840319 18947 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "515eaf1692704fb2b4545919cee00426" candidate_uuid: "1f482de249fc4d21ad5da40eed04ef28" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb648d7294dc41009b425c5289ad24e8"
I20260618 22:21:50.840448 18947 raft_consensus.cc:3037] T 515eaf1692704fb2b4545919cee00426 P bb648d7294dc41009b425c5289ad24e8 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:21:50.840447 18679 raft_consensus.cc:3037] T 515eaf1692704fb2b4545919cee00426 P cea947528dbd428990dc14e73f624272 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:21:50.841681 18679 raft_consensus.cc:2454] T 515eaf1692704fb2b4545919cee00426 P cea947528dbd428990dc14e73f624272 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f482de249fc4d21ad5da40eed04ef28 in term 1.
I20260618 22:21:50.841681 18947 raft_consensus.cc:2454] T 515eaf1692704fb2b4545919cee00426 P bb648d7294dc41009b425c5289ad24e8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1f482de249fc4d21ad5da40eed04ef28 in term 1.
I20260618 22:21:50.842010 18749 leader_election.cc:302] T 515eaf1692704fb2b4545919cee00426 P 1f482de249fc4d21ad5da40eed04ef28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f482de249fc4d21ad5da40eed04ef28, bb648d7294dc41009b425c5289ad24e8; no voters:
I20260618 22:21:50.842166 19018 raft_consensus.cc:2781] T 515eaf1692704fb2b4545919cee00426 P 1f482de249fc4d21ad5da40eed04ef28 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:21:50.842376 19018 raft_consensus.cc:686] T 515eaf1692704fb2b4545919cee00426 P 1f482de249fc4d21ad5da40eed04ef28 [term 1 LEADER]: Becoming Leader. State: Replica: 1f482de249fc4d21ad5da40eed04ef28, State: Running, Role: LEADER
I20260618 22:21:50.842687 19018 consensus_queue.cc:237] T 515eaf1692704fb2b4545919cee00426 P 1f482de249fc4d21ad5da40eed04ef28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb648d7294dc41009b425c5289ad24e8" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 44501 } } peers { permanent_uuid: "cea947528dbd428990dc14e73f624272" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33733 } } peers { permanent_uuid: "1f482de249fc4d21ad5da40eed04ef28" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43769 } }
I20260618 22:21:50.845621 18531 catalog_manager.cc:5495] T 515eaf1692704fb2b4545919cee00426 P 1f482de249fc4d21ad5da40eed04ef28 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1f482de249fc4d21ad5da40eed04ef28 (127.0.98.66). New cstate: current_term: 1 leader_uuid: "1f482de249fc4d21ad5da40eed04ef28" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bb648d7294dc41009b425c5289ad24e8" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 44501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cea947528dbd428990dc14e73f624272" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33733 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f482de249fc4d21ad5da40eed04ef28" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43769 } health_report { overall_health: HEALTHY } } }
I20260618 22:21:50.865736 19009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:50.862351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49126 (local address 127.0.98.66:43769)
0618 22:21:50.862495 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:50.862499 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:50.862514 (+ 15us) server_negotiation.cc:407] Connection header received
0618 22:21:50.862581 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:50.862583 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:50.862634 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:50.862748 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:21:50.862972 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:50.863494 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:50.864378 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:50.864606 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:50.864693 (+ 87us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:50.864979 (+ 286us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:21:50.865364 (+ 385us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:21:50.865474 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:50.865599 (+ 125us) server_negotiation.cc:299] Negotiation successful
0618 22:21:50.865630 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260618 22:21:50.871409 18947 raft_consensus.cc:1261] T 515eaf1692704fb2b4545919cee00426 P bb648d7294dc41009b425c5289ad24e8 [term 1 FOLLOWER]: Refusing update from remote peer 1f482de249fc4d21ad5da40eed04ef28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:21:50.871409 18679 raft_consensus.cc:1261] T 515eaf1692704fb2b4545919cee00426 P cea947528dbd428990dc14e73f624272 [term 1 FOLLOWER]: Refusing update from remote peer 1f482de249fc4d21ad5da40eed04ef28: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:21:50.872059 19018 consensus_queue.cc:1035] T 515eaf1692704fb2b4545919cee00426 P 1f482de249fc4d21ad5da40eed04ef28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb648d7294dc41009b425c5289ad24e8" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 44501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:21:50.872246 19021 consensus_queue.cc:1035] T 515eaf1692704fb2b4545919cee00426 P 1f482de249fc4d21ad5da40eed04ef28 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cea947528dbd428990dc14e73f624272" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 33733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:21:50.881289 19026 mvcc.cc:205] Tried to move back new op lower bound from 7298340089322180608 to 7298340089213997056. Current Snapshot: MvccSnapshot[applied={T|T < 7298340089322180608}]
I20260618 22:21:50.887876 18534 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58590:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:21:50.888054 18534 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58590:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:21:50.890441 18534 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 555a52e41592429384d42cc1fe9ebcbc: Sending DeleteTablet for 3 replicas of tablet 515eaf1692704fb2b4545919cee00426
I20260618 22:21:50.891149 18927 tablet_service.cc:1514] Processing DeleteTablet for tablet 515eaf1692704fb2b4545919cee00426 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:21:50 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:59592
I20260618 22:21:50.891297 18659 tablet_service.cc:1514] Processing DeleteTablet for tablet 515eaf1692704fb2b4545919cee00426 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:21:50 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:57968
I20260618 22:21:50.891710 19035 tablet_replica.cc:330] T 515eaf1692704fb2b4545919cee00426 P cea947528dbd428990dc14e73f624272: stopping tablet replica
I20260618 22:21:50.891824 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 18596
I20260618 22:21:50.892000 19035 raft_consensus.cc:2229] T 515eaf1692704fb2b4545919cee00426 P cea947528dbd428990dc14e73f624272 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:21:50.892019 19034 tablet_replica.cc:330] T 515eaf1692704fb2b4545919cee00426 P bb648d7294dc41009b425c5289ad24e8: stopping tablet replica
I20260618 22:21:50.892367 19034 raft_consensus.cc:2229] T 515eaf1692704fb2b4545919cee00426 P bb648d7294dc41009b425c5289ad24e8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:21:50.892488 18793 tablet_service.cc:1514] Processing DeleteTablet for tablet 515eaf1692704fb2b4545919cee00426 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:21:50 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:49112
I20260618 22:21:50.892724 19034 raft_consensus.cc:2258] T 515eaf1692704fb2b4545919cee00426 P bb648d7294dc41009b425c5289ad24e8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:21:50.894150 19034 ts_tablet_manager.cc:1905] T 515eaf1692704fb2b4545919cee00426 P bb648d7294dc41009b425c5289ad24e8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:21:50.895131 19036 tablet_replica.cc:330] T 515eaf1692704fb2b4545919cee00426 P 1f482de249fc4d21ad5da40eed04ef28: stopping tablet replica
I20260618 22:21:50.895386 19036 raft_consensus.cc:2229] T 515eaf1692704fb2b4545919cee00426 P 1f482de249fc4d21ad5da40eed04ef28 [term 1 LEADER]: Raft consensus shutting down.
I20260618 22:21:50.895737 19036 raft_consensus.cc:2258] T 515eaf1692704fb2b4545919cee00426 P 1f482de249fc4d21ad5da40eed04ef28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:21:50.897054 19036 ts_tablet_manager.cc:1905] T 515eaf1692704fb2b4545919cee00426 P 1f482de249fc4d21ad5da40eed04ef28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:21:50.897184 19034 ts_tablet_manager.cc:1918] T 515eaf1692704fb2b4545919cee00426 P bb648d7294dc41009b425c5289ad24e8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:21:50.897305 19034 log.cc:1196] T 515eaf1692704fb2b4545919cee00426 P bb648d7294dc41009b425c5289ad24e8: Deleting WAL directory at /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/minicluster-data/ts-2/wal/wals/515eaf1692704fb2b4545919cee00426
I20260618 22:21:50.897714 19034 ts_tablet_manager.cc:1939] T 515eaf1692704fb2b4545919cee00426 P bb648d7294dc41009b425c5289ad24e8: Deleting consensus metadata
I20260618 22:21:50.898995 18521 catalog_manager.cc:4841] TS bb648d7294dc41009b425c5289ad24e8 (127.0.98.67:44501): tablet 515eaf1692704fb2b4545919cee00426 (table test-table [id=fe1bbbf5ca8845e6b2f94983465914a2]) successfully deleted
I20260618 22:21:50.899681 19036 ts_tablet_manager.cc:1918] T 515eaf1692704fb2b4545919cee00426 P 1f482de249fc4d21ad5da40eed04ef28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:21:50.899786 19036 log.cc:1196] T 515eaf1692704fb2b4545919cee00426 P 1f482de249fc4d21ad5da40eed04ef28: Deleting WAL directory at /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781821065380026-393-0/minicluster-data/ts-1/wal/wals/515eaf1692704fb2b4545919cee00426
I20260618 22:21:50.900048 19036 ts_tablet_manager.cc:1939] T 515eaf1692704fb2b4545919cee00426 P 1f482de249fc4d21ad5da40eed04ef28: Deleting consensus metadata
I20260618 22:21:50.900921 18521 catalog_manager.cc:4841] TS 1f482de249fc4d21ad5da40eed04ef28 (127.0.98.66:43769): tablet 515eaf1692704fb2b4545919cee00426 (table test-table [id=fe1bbbf5ca8845e6b2f94983465914a2]) successfully deleted
I20260618 22:21:50.905050 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 18730
W20260618 22:21:50.905179 18521 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.65:33733 (error 108)
I20260618 22:21:50.910024 19008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:50.906654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:33733 (local address 127.0.0.1:57974)
0618 22:21:50.909827 (+ 3173us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:50.909934 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.65:33733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3081}
W20260618 22:21:50.910192 18521 catalog_manager.cc:4584] TS cea947528dbd428990dc14e73f624272 (127.0.98.65:33733): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 515eaf1692704fb2b4545919cee00426: Network error: Client connection negotiation failed: client connection to 127.0.98.65:33733: connect: Connection refused (error 111)
I20260618 22:21:50.910269 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 18864
I20260618 22:21:50.915318 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 18502
2026-06-18T22:21:50Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2504 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-18T22:21:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:21:50Z Disabled control of system clock
I20260618 22:21:50.944283 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821065380026-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821065380026-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821065380026-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:46469
--webserver_interface=127.0.98.126
--webserver_port=0
--builtin_ntp_servers=127.0.98.84:35863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:46469
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260618 22:21:51.051721 19041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:21:51.051962 19041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:21:51.051996 19041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:21:51.055919 19041 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:21:51.055968 19041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:21:51.055984 19041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:21:51.056000 19041 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:21:51.056015 19041 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:21:51.060648 19041 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:35863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821065380026-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821065380026-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:46469
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:46469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19041
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821065380026-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:21:51.061547 19041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:21:51.062865 19041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260618 22:21:51.071128 19041 server_base.cc:1034] running on GCE node
W20260618 22:21:51.071422 19049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:21:51.071373 19047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:21:51.071389 19046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:21:51.072181 19041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:21:51.073189 19041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:21:51.074373 19041 hybrid_clock.cc:648] HybridClock initialized: now 1781821311074346 us; error 47 us; skew 500 ppm
I20260618 22:21:51.076179 19041 webserver.cc:460] Webserver started at http://127.0.98.126:41575/ using document root <none> and password file <none>
I20260618 22:21:51.076753 19041 fs_manager.cc:359] Metadata directory not provided
I20260618 22:21:51.076845 19041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:21:51.077077 19041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:21:51.079154 19041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821065380026-393-0/minicluster-data/master-0/data/instance:
uuid: "486161f1d2a74f808c2590417ea08793"
format_stamp: "Formatted at 2026-06-18 22:21:51 on dist-test-slave-g6r4"
server_key: "6bfbcb31fa10fa8cd570d0c38235739e"
server_key_iv: "5ecbe80b99483b0618b6b68c21ecbe19"
server_key_version: "encryptionkey@0"
I20260618 22:21:51.079706 19041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821065380026-393-0/minicluster-data/master-0/wal/instance:
uuid: "486161f1d2a74f808c2590417ea08793"
format_stamp: "Formatted at 2026-06-18 22:21:51 on dist-test-slave-g6r4"
server_key: "6bfbcb31fa10fa8cd570d0c38235739e"
server_key_iv: "5ecbe80b99483b0618b6b68c21ecbe19"
server_key_version: "encryptionkey@0"
I20260618 22:21:51.083523 19041 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260618 22:21:51.085817 19055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:51.087162 19041 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:21:51.087311 19041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821065380026-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821065380026-393-0/minicluster-data/master-0/wal
uuid: "486161f1d2a74f808c2590417ea08793"
format_stamp: "Formatted at 2026-06-18 22:21:51 on dist-test-slave-g6r4"
server_key: "6bfbcb31fa10fa8cd570d0c38235739e"
server_key_iv: "5ecbe80b99483b0618b6b68c21ecbe19"
server_key_version: "encryptionkey@0"
I20260618 22:21:51.087417 19041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821065380026-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821065380026-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821065380026-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:21:51.101595 19041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:21:51.102248 19041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:21:51.102442 19041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:21:51.109169 19041 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:46469
I20260618 22:21:51.109202 19107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:46469 every 8 connection(s)
I20260618 22:21:51.110102 19041 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
I20260618 22:21:51.112959 19108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:21:51.118052 19108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 486161f1d2a74f808c2590417ea08793: Bootstrap starting.
I20260618 22:21:51.119223 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 19041
I20260618 22:21:51.119310 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821065380026-393-0/minicluster-data/master-0/wal/instance
I20260618 22:21:51.119609 393 external_mini_cluster.cc:1428] Setting key 41d1e11bd03ad0a6ff5afae9a81f59b4
I20260618 22:21:51.120549 19108 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 486161f1d2a74f808c2590417ea08793: Neither blocks nor log segments found. Creating new log.
I20260618 22:21:51.121260 19108 log.cc:824] T 00000000000000000000000000000000 P 486161f1d2a74f808c2590417ea08793: Log is configured to *not* fsync() on all Append() calls
I20260618 22:21:51.123306 19108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 486161f1d2a74f808c2590417ea08793: No bootstrap required, opened a new log
I20260618 22:21:51.127357 19108 raft_consensus.cc:348] T 00000000000000000000000000000000 P 486161f1d2a74f808c2590417ea08793 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486161f1d2a74f808c2590417ea08793" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46469 } }
I20260618 22:21:51.127621 19108 raft_consensus.cc:374] T 00000000000000000000000000000000 P 486161f1d2a74f808c2590417ea08793 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:21:51.127686 19108 raft_consensus.cc:729] T 00000000000000000000000000000000 P 486161f1d2a74f808c2590417ea08793 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 486161f1d2a74f808c2590417ea08793, State: Initialized, Role: FOLLOWER
I20260618 22:21:51.128096 19108 consensus_queue.cc:260] T 00000000000000000000000000000000 P 486161f1d2a74f808c2590417ea08793 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486161f1d2a74f808c2590417ea08793" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46469 } }
I20260618 22:21:51.128221 19108 raft_consensus.cc:388] T 00000000000000000000000000000000 P 486161f1d2a74f808c2590417ea08793 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:21:51.128306 19108 raft_consensus.cc:482] T 00000000000000000000000000000000 P 486161f1d2a74f808c2590417ea08793 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:21:51.128430 19108 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 486161f1d2a74f808c2590417ea08793 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:21:51.128944 19111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:51.120687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58074 (local address 127.0.98.126:46469)
0618 22:21:51.121309 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:51.121322 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:51.121355 (+ 33us) server_negotiation.cc:407] Connection header received
0618 22:21:51.122205 (+ 850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:51.122224 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:51.122532 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:51.122798 (+ 266us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:21:51.123253 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:51.124254 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:51.125463 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:51.125972 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:51.126095 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:51.126955 (+ 860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:21:51.126960 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:21:51.126972 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:21:51.127355 (+ 383us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:21:51.127407 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:21:51.127426 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:21:51.127500 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:51.127795 (+ 295us) server_negotiation.cc:299] Negotiation successful
0618 22:21:51.128093 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":132,"threads_started":1}
I20260618 22:21:51.129334 19108 raft_consensus.cc:504] T 00000000000000000000000000000000 P 486161f1d2a74f808c2590417ea08793 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486161f1d2a74f808c2590417ea08793" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46469 } }
I20260618 22:21:51.129690 19108 leader_election.cc:302] T 00000000000000000000000000000000 P 486161f1d2a74f808c2590417ea08793 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 486161f1d2a74f808c2590417ea08793; no voters:
I20260618 22:21:51.130030 19108 leader_election.cc:288] T 00000000000000000000000000000000 P 486161f1d2a74f808c2590417ea08793 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:21:51.130097 19113 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 486161f1d2a74f808c2590417ea08793 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:21:51.130333 19113 raft_consensus.cc:686] T 00000000000000000000000000000000 P 486161f1d2a74f808c2590417ea08793 [term 1 LEADER]: Becoming Leader. State: Replica: 486161f1d2a74f808c2590417ea08793, State: Running, Role: LEADER
I20260618 22:21:51.130635 19113 consensus_queue.cc:237] T 00000000000000000000000000000000 P 486161f1d2a74f808c2590417ea08793 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486161f1d2a74f808c2590417ea08793" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46469 } }
I20260618 22:21:51.131201 19108 sys_catalog.cc:564] T 00000000000000000000000000000000 P 486161f1d2a74f808c2590417ea08793 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:21:51.132275 19114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 486161f1d2a74f808c2590417ea08793 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "486161f1d2a74f808c2590417ea08793" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486161f1d2a74f808c2590417ea08793" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46469 } } }
I20260618 22:21:51.132418 19114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 486161f1d2a74f808c2590417ea08793 [sys.catalog]: This master's current role is: LEADER
I20260618 22:21:51.132275 19115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 486161f1d2a74f808c2590417ea08793 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 486161f1d2a74f808c2590417ea08793. Latest consensus state: current_term: 1 leader_uuid: "486161f1d2a74f808c2590417ea08793" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "486161f1d2a74f808c2590417ea08793" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46469 } } }
I20260618 22:21:51.132745 19115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 486161f1d2a74f808c2590417ea08793 [sys.catalog]: This master's current role is: LEADER
I20260618 22:21:51.132786 19122 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:21:51.135715 19122 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:21:51.140841 19122 catalog_manager.cc:1347] Generated new cluster ID: b3474d15ce6b4177b7695a809197189b
I20260618 22:21:51.140903 19122 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:21:51.166693 19122 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:21:51.167811 19122 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:21:51.176506 19122 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 486161f1d2a74f808c2590417ea08793: Generated new TSK 0
I20260618 22:21:51.177182 19122 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:21:51.190088 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821065380026-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821065380026-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821065380026-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:46469
--builtin_ntp_servers=127.0.98.84:35863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260618 22:21:51.298998 19132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:21:51.299240 19132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:21:51.299276 19132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:21:51.303117 19132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:21:51.303164 19132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:21:51.303215 19132 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260618 22:21:51.307824 19132 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:35863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821065380026-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821065380026-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:46469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821065380026-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:21:51.308763 19132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:21:51.310086 19132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:21:51.319432 19137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:21:51.319482 19140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:21:51.319641 19138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:21:51.319510 19132 server_base.cc:1034] running on GCE node
I20260618 22:21:51.320055 19132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:21:51.320662 19132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:21:51.321820 19132 hybrid_clock.cc:648] HybridClock initialized: now 1781821311321786 us; error 47 us; skew 500 ppm
I20260618 22:21:51.323585 19132 webserver.cc:460] Webserver started at http://127.0.98.65:38699/ using document root <none> and password file <none>
I20260618 22:21:51.324160 19132 fs_manager.cc:359] Metadata directory not provided
I20260618 22:21:51.324222 19132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:21:51.324469 19132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:21:51.326584 19132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821065380026-393-0/minicluster-data/ts-0/data/instance:
uuid: "cd629930df894fe4baea4619beea75d5"
format_stamp: "Formatted at 2026-06-18 22:21:51 on dist-test-slave-g6r4"
server_key: "5a43be164ebc51754f3d8593e73f9071"
server_key_iv: "62ec10867695153c59c2792208821da6"
server_key_version: "encryptionkey@0"
I20260618 22:21:51.327139 19132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821065380026-393-0/minicluster-data/ts-0/wal/instance:
uuid: "cd629930df894fe4baea4619beea75d5"
format_stamp: "Formatted at 2026-06-18 22:21:51 on dist-test-slave-g6r4"
server_key: "5a43be164ebc51754f3d8593e73f9071"
server_key_iv: "62ec10867695153c59c2792208821da6"
server_key_version: "encryptionkey@0"
I20260618 22:21:51.330731 19132 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:21:51.333212 19146 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:51.334430 19132 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:21:51.334545 19132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821065380026-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821065380026-393-0/minicluster-data/ts-0/wal
uuid: "cd629930df894fe4baea4619beea75d5"
format_stamp: "Formatted at 2026-06-18 22:21:51 on dist-test-slave-g6r4"
server_key: "5a43be164ebc51754f3d8593e73f9071"
server_key_iv: "62ec10867695153c59c2792208821da6"
server_key_version: "encryptionkey@0"
I20260618 22:21:51.334656 19132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821065380026-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821065380026-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821065380026-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:21:51.355458 19132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:21:51.356174 19132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:21:51.356377 19132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:21:51.356966 19132 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:21:51.357884 19132 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:21:51.357946 19132 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:51.357987 19132 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:21:51.358027 19132 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:51.366933 19132 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:46365
I20260618 22:21:51.366961 19259 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:46365 every 8 connection(s)
I20260618 22:21:51.367800 19132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
I20260618 22:21:51.376233 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 19132
I20260618 22:21:51.376387 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821065380026-393-0/minicluster-data/ts-0/wal/instance
I20260618 22:21:51.376760 393 external_mini_cluster.cc:1428] Setting key 7069943c64967b5f6517afb9cd15ba5b
I20260618 22:21:51.377053 19111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:51.369552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:40237 (local address 127.0.98.126:46469)
0618 22:21:51.369710 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:51.369715 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:51.370826 (+ 1111us) server_negotiation.cc:407] Connection header received
0618 22:21:51.372145 (+ 1319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:51.372148 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:51.372193 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:51.372268 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:21:51.373182 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:51.373699 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:51.374950 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:51.375178 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:51.375261 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:51.376378 (+ 1117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:21:51.376382 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:21:51.376385 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:21:51.376398 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:21:51.376426 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:21:51.376435 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:21:51.376527 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:51.376762 (+ 235us) server_negotiation.cc:299] Negotiation successful
0618 22:21:51.376859 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260618 22:21:51.377988 19262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:51.370028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:46469 (local address 127.0.98.65:40237)
0618 22:21:51.370680 (+ 652us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:51.370720 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0618 22:21:51.371950 (+ 1230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:21:51.372435 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:21:51.372441 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:21:51.372522 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:21:51.373057 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:51.373070 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:51.373829 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:51.373832 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:51.374845 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:51.374854 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:51.375305 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:51.375308 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:51.375455 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:51.376084 (+ 629us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 22:21:51.376085 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:21:51.376132 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 22:21:51.376222 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:21:51.376543 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:21:51.376555 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:21:51.376562 (+ 7us) client_negotiation.cc:769] Sending connection context
0618 22:21:51.376717 (+ 155us) client_negotiation.cc:240] Negotiation successful
0618 22:21:51.376997 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":104,"threads_started":1}
I20260618 22:21:51.378973 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821065380026-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821065380026-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821065380026-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:46469
--builtin_ntp_servers=127.0.98.84:35863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260618 22:21:51.379357 19260 heartbeater.cc:347] Connected to a master server at 127.0.98.126:46469
I20260618 22:21:51.379612 19260 heartbeater.cc:464] Registering TS with master...
I20260618 22:21:51.380122 19260 heartbeater.cc:511] Master 127.0.98.126:46469 requested a full tablet report, sending...
I20260618 22:21:51.381608 19071 ts_manager.cc:194] Registered new tserver with Master: cd629930df894fe4baea4619beea75d5 (127.0.98.65:46365)
I20260618 22:21:51.382791 19071 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.65:40237
W20260618 22:21:51.489122 19263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:21:51.489387 19263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:21:51.489426 19263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:21:51.493688 19263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:21:51.493747 19263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:21:51.493803 19263 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260618 22:21:51.498610 19263 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:35863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821065380026-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821065380026-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:46469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19263
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821065380026-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:21:51.499699 19263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:21:51.501063 19263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:21:51.510502 19268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:21:51.510674 19263 server_base.cc:1034] running on GCE node
W20260618 22:21:51.510553 19271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:21:51.510532 19269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:21:51.511183 19263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:21:51.511878 19263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:21:51.513058 19263 hybrid_clock.cc:648] HybridClock initialized: now 1781821311513040 us; error 29 us; skew 500 ppm
I20260618 22:21:51.514942 19263 webserver.cc:460] Webserver started at http://127.0.98.66:38479/ using document root <none> and password file <none>
I20260618 22:21:51.515630 19263 fs_manager.cc:359] Metadata directory not provided
I20260618 22:21:51.515719 19263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:21:51.515929 19263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:21:51.517961 19263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821065380026-393-0/minicluster-data/ts-1/data/instance:
uuid: "1160f810d89648b6abea53a588f83fe4"
format_stamp: "Formatted at 2026-06-18 22:21:51 on dist-test-slave-g6r4"
server_key: "c8e1975ac38a67527f0a6da4d3415dfa"
server_key_iv: "787db9cb5e165942394cef0165ad3dc1"
server_key_version: "encryptionkey@0"
I20260618 22:21:51.518476 19263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821065380026-393-0/minicluster-data/ts-1/wal/instance:
uuid: "1160f810d89648b6abea53a588f83fe4"
format_stamp: "Formatted at 2026-06-18 22:21:51 on dist-test-slave-g6r4"
server_key: "c8e1975ac38a67527f0a6da4d3415dfa"
server_key_iv: "787db9cb5e165942394cef0165ad3dc1"
server_key_version: "encryptionkey@0"
I20260618 22:21:51.521997 19263 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260618 22:21:51.524402 19277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:51.525671 19263 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260618 22:21:51.525771 19263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821065380026-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821065380026-393-0/minicluster-data/ts-1/wal
uuid: "1160f810d89648b6abea53a588f83fe4"
format_stamp: "Formatted at 2026-06-18 22:21:51 on dist-test-slave-g6r4"
server_key: "c8e1975ac38a67527f0a6da4d3415dfa"
server_key_iv: "787db9cb5e165942394cef0165ad3dc1"
server_key_version: "encryptionkey@0"
I20260618 22:21:51.525859 19263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821065380026-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821065380026-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821065380026-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:21:51.545521 19263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:21:51.546345 19263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:21:51.546499 19263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:21:51.547030 19263 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:21:51.547930 19263 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:21:51.547977 19263 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:51.548017 19263 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:21:51.548032 19263 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:51.557070 19263 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:43495
I20260618 22:21:51.557101 19390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:43495 every 8 connection(s)
I20260618 22:21:51.558050 19263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
I20260618 22:21:51.565614 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 19263
I20260618 22:21:51.565775 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821065380026-393-0/minicluster-data/ts-1/wal/instance
I20260618 22:21:51.566115 393 external_mini_cluster.cc:1428] Setting key e2cbbd70e9a04d785520478ef96b77d0
I20260618 22:21:51.567346 19111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:51.559867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:33461 (local address 127.0.98.126:46469)
0618 22:21:51.560017 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:51.560021 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:51.561045 (+ 1024us) server_negotiation.cc:407] Connection header received
0618 22:21:51.562528 (+ 1483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:51.562532 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:51.562592 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:51.562677 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:21:51.563568 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:51.564102 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:51.565249 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:51.565480 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:51.565594 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:51.566879 (+ 1285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:21:51.566883 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:21:51.566886 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:21:51.566900 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:21:51.566922 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:21:51.566932 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:21:51.567015 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:51.567160 (+ 145us) server_negotiation.cc:299] Negotiation successful
0618 22:21:51.567211 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260618 22:21:51.568188 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821065380026-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821065380026-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821065380026-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:46469
--builtin_ntp_servers=127.0.98.84:35863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260618 22:21:51.568210 19393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:51.560257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:46469 (local address 127.0.98.66:33461)
0618 22:21:51.560887 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:51.560927 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0618 22:21:51.562294 (+ 1367us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:21:51.562865 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:21:51.562872 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:21:51.562956 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:21:51.563424 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:51.563436 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:51.564245 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:51.564255 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:51.565121 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:51.565130 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:51.565641 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:51.565647 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:51.565881 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:51.566553 (+ 672us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 22:21:51.566555 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:21:51.566594 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 22:21:51.566696 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:21:51.567023 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:21:51.567031 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:21:51.567035 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 22:21:51.567137 (+ 102us) client_negotiation.cc:240] Negotiation successful
0618 22:21:51.567384 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":104,"threads_started":1}
I20260618 22:21:51.569336 19391 heartbeater.cc:347] Connected to a master server at 127.0.98.126:46469
I20260618 22:21:51.569598 19391 heartbeater.cc:464] Registering TS with master...
I20260618 22:21:51.570187 19391 heartbeater.cc:511] Master 127.0.98.126:46469 requested a full tablet report, sending...
I20260618 22:21:51.571197 19071 ts_manager.cc:194] Registered new tserver with Master: 1160f810d89648b6abea53a588f83fe4 (127.0.98.66:43495)
I20260618 22:21:51.571763 19071 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.66:33461
W20260618 22:21:51.679520 19394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:21:51.679766 19394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:21:51.679802 19394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:21:51.683926 19394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:21:51.683976 19394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:21:51.684060 19394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260618 22:21:51.688742 19394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:35863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821065380026-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821065380026-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:46469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821065380026-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:21:51.689738 19394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:21:51.691129 19394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260618 22:21:51.700553 19394 server_base.cc:1034] running on GCE node
W20260618 22:21:51.700582 19400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:21:51.700667 19402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:21:51.700690 19399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:21:51.701152 19394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:21:51.701756 19394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:21:51.702934 19394 hybrid_clock.cc:648] HybridClock initialized: now 1781821311702915 us; error 31 us; skew 500 ppm
I20260618 22:21:51.704792 19394 webserver.cc:460] Webserver started at http://127.0.98.67:34031/ using document root <none> and password file <none>
I20260618 22:21:51.705423 19394 fs_manager.cc:359] Metadata directory not provided
I20260618 22:21:51.705489 19394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:21:51.705698 19394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:21:51.708673 19394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821065380026-393-0/minicluster-data/ts-2/data/instance:
uuid: "2c4af571f140401abccc99931b958257"
format_stamp: "Formatted at 2026-06-18 22:21:51 on dist-test-slave-g6r4"
server_key: "4ed76cc95f5a9c3950313edfd93cd51e"
server_key_iv: "13ecf7dd4b46dc227c4462ba86a28a39"
server_key_version: "encryptionkey@0"
I20260618 22:21:51.709378 19394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821065380026-393-0/minicluster-data/ts-2/wal/instance:
uuid: "2c4af571f140401abccc99931b958257"
format_stamp: "Formatted at 2026-06-18 22:21:51 on dist-test-slave-g6r4"
server_key: "4ed76cc95f5a9c3950313edfd93cd51e"
server_key_iv: "13ecf7dd4b46dc227c4462ba86a28a39"
server_key_version: "encryptionkey@0"
I20260618 22:21:51.713074 19394 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260618 22:21:51.715550 19408 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:51.716806 19394 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260618 22:21:51.716989 19394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821065380026-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821065380026-393-0/minicluster-data/ts-2/wal
uuid: "2c4af571f140401abccc99931b958257"
format_stamp: "Formatted at 2026-06-18 22:21:51 on dist-test-slave-g6r4"
server_key: "4ed76cc95f5a9c3950313edfd93cd51e"
server_key_iv: "13ecf7dd4b46dc227c4462ba86a28a39"
server_key_version: "encryptionkey@0"
I20260618 22:21:51.717108 19394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821065380026-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821065380026-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821065380026-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:21:51.736305 19394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:21:51.736989 19394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:21:51.737180 19394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:21:51.737741 19394 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:21:51.738724 19394 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:21:51.738794 19394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:51.738862 19394 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:21:51.738898 19394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:51.748097 19394 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:33719
I20260618 22:21:51.748152 19521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:33719 every 8 connection(s)
I20260618 22:21:51.749017 19394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
I20260618 22:21:51.754879 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 19394
I20260618 22:21:51.755000 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821065380026-393-0/minicluster-data/ts-2/wal/instance
I20260618 22:21:51.755322 393 external_mini_cluster.cc:1428] Setting key 64fd46e37570b6137a1b14f5f316ff34
I20260618 22:21:51.758517 19111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:51.750811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:52677 (local address 127.0.98.126:46469)
0618 22:21:51.750964 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:51.750968 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:51.752026 (+ 1058us) server_negotiation.cc:407] Connection header received
0618 22:21:51.753320 (+ 1294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:51.753323 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:51.753369 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:51.753445 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:21:51.754412 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:51.755178 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:51.756532 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:51.756812 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:51.756928 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:51.758045 (+ 1117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:21:51.758048 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:21:51.758051 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:21:51.758063 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:21:51.758089 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:21:51.758097 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:21:51.758176 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:51.758327 (+ 151us) server_negotiation.cc:299] Negotiation successful
0618 22:21:51.758384 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260618 22:21:51.759315 19524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:51.751253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:46469 (local address 127.0.98.67:52677)
0618 22:21:51.751877 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:51.751916 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0618 22:21:51.753125 (+ 1209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:21:51.753618 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:21:51.753624 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:21:51.753708 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:21:51.754293 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:51.754306 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:51.755315 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:51.755318 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:51.756400 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:51.756409 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:51.756960 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:51.756964 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:51.757128 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:51.757740 (+ 612us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 22:21:51.757742 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:21:51.757779 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 22:21:51.757837 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:21:51.758178 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:21:51.758189 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:21:51.758196 (+ 7us) client_negotiation.cc:769] Sending connection context
0618 22:21:51.758327 (+ 131us) client_negotiation.cc:240] Negotiation successful
0618 22:21:51.758602 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":104,"threads_started":1}
I20260618 22:21:51.760425 19522 heartbeater.cc:347] Connected to a master server at 127.0.98.126:46469
I20260618 22:21:51.760628 19522 heartbeater.cc:464] Registering TS with master...
I20260618 22:21:51.761125 19522 heartbeater.cc:511] Master 127.0.98.126:46469 requested a full tablet report, sending...
I20260618 22:21:51.762130 19071 ts_manager.cc:194] Registered new tserver with Master: 2c4af571f140401abccc99931b958257 (127.0.98.67:33719)
I20260618 22:21:51.762667 19071 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.67:52677
I20260618 22:21:51.768925 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 22:21:51.774338 19111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:51.770846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58088 (local address 127.0.98.126:46469)
0618 22:21:51.770962 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:51.770966 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:51.771092 (+ 126us) server_negotiation.cc:407] Connection header received
0618 22:21:51.771188 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:51.771190 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:51.771226 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:51.771291 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:21:51.771525 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:51.772019 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:51.772782 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:51.772988 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:51.773061 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:51.773844 (+ 783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:21:51.773847 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:21:51.773849 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:21:51.773860 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:21:51.773878 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:21:51.773885 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:21:51.774053 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:51.774197 (+ 144us) server_negotiation.cc:299] Negotiation successful
0618 22:21:51.774235 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260618 22:21:51.777190 19071 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:58088:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 22:21:51.780278 19071 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:21:51.795071 19537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:51.789370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51778 (local address 127.0.98.66:43495)
0618 22:21:51.790027 (+ 657us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:51.790039 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:51.790062 (+ 23us) server_negotiation.cc:407] Connection header received
0618 22:21:51.790132 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:51.790138 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:51.790368 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:51.790520 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:21:51.790896 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:51.791854 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:51.793568 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:51.794329 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:51.794466 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:51.794673 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:51.794798 (+ 125us) server_negotiation.cc:299] Negotiation successful
0618 22:21:51.794894 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":111,"threads_started":1}
I20260618 22:21:51.795059 19535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:51.788730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:43495 (local address 127.0.0.1:51778)
0618 22:21:51.789303 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:51.789371 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0618 22:21:51.789514 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:21:51.790640 (+ 1126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:21:51.790642 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:21:51.790657 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:21:51.790743 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:51.790748 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:51.791958 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:51.791964 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:51.793432 (+ 1468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:51.793444 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:51.794466 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:51.794470 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:51.794598 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:51.794637 (+ 39us) client_negotiation.cc:769] Sending connection context
0618 22:21:51.794760 (+ 123us) client_negotiation.cc:240] Negotiation successful
0618 22:21:51.794888 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"mutex_wait_us":102,"thread_start_us":124,"threads_started":1}
I20260618 22:21:51.795338 19538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:51.789392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40852 (local address 127.0.98.67:33719)
0618 22:21:51.789783 (+ 391us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:51.789797 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:51.790153 (+ 356us) server_negotiation.cc:407] Connection header received
0618 22:21:51.790406 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:51.790413 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:51.790597 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:51.790718 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:21:51.791112 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:51.792311 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:51.793784 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:51.794543 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:51.794634 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:51.794771 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:51.795065 (+ 294us) server_negotiation.cc:299] Negotiation successful
0618 22:21:51.795161 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":146,"threads_started":1}
I20260618 22:21:51.795531 19533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:51.788599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:46365 (local address 127.0.0.1:60846)
0618 22:21:51.789679 (+ 1080us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:51.789695 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 22:21:51.789797 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:21:51.790512 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:21:51.790518 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:21:51.790540 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:21:51.790697 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:51.790705 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:51.792253 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:51.792257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:51.793462 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:51.793471 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:51.794761 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:51.794765 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:51.794855 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:51.794879 (+ 24us) client_negotiation.cc:769] Sending connection context
0618 22:21:51.794950 (+ 71us) client_negotiation.cc:240] Negotiation successful
0618 22:21:51.795036 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":426,"mutex_wait_us":31,"thread_start_us":83,"threads_started":1}
I20260618 22:21:51.795737 19536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:51.789019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:33719 (local address 127.0.0.1:40852)
0618 22:21:51.790023 (+ 1004us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:51.790036 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:21:51.790255 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:21:51.790896 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:21:51.790899 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:21:51.790916 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:21:51.790988 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:51.790994 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:51.792422 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:51.792428 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:51.793656 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:51.793667 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:51.794819 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:51.794822 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:51.794902 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:51.794936 (+ 34us) client_negotiation.cc:769] Sending connection context
0618 22:21:51.795518 (+ 582us) client_negotiation.cc:240] Negotiation successful
0618 22:21:51.795600 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":939,"mutex_wait_us":22,"thread_start_us":90,"threads_started":1}
I20260618 22:21:51.796118 19534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:51.788722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60846 (local address 127.0.98.65:46365)
0618 22:21:51.789514 (+ 792us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:51.789527 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:51.789807 (+ 280us) server_negotiation.cc:407] Connection header received
0618 22:21:51.789935 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:51.789941 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:51.790162 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:51.790321 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:21:51.790829 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:51.792137 (+ 1308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:51.793603 (+ 1466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:51.794389 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:51.794496 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:51.794695 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:51.795564 (+ 869us) server_negotiation.cc:299] Negotiation successful
0618 22:21:51.795956 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":678,"thread_start_us":116,"threads_started":1}
I20260618 22:21:51.798307 19325 tablet_service.cc:1467] Processing CreateTablet for tablet 7573621589a64dd2b458ec974881424f (DEFAULT_TABLE table=test-table [id=8081a01e9e5a445d8882f3f1e2816bc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:21:51.798689 19194 tablet_service.cc:1467] Processing CreateTablet for tablet 7573621589a64dd2b458ec974881424f (DEFAULT_TABLE table=test-table [id=8081a01e9e5a445d8882f3f1e2816bc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:21:51.798835 19456 tablet_service.cc:1467] Processing CreateTablet for tablet 7573621589a64dd2b458ec974881424f (DEFAULT_TABLE table=test-table [id=8081a01e9e5a445d8882f3f1e2816bc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:21:51.799443 19325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7573621589a64dd2b458ec974881424f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:21:51.799733 19194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7573621589a64dd2b458ec974881424f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:21:51.799789 19456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7573621589a64dd2b458ec974881424f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:21:51.806118 19539 tablet_bootstrap.cc:492] T 7573621589a64dd2b458ec974881424f P cd629930df894fe4baea4619beea75d5: Bootstrap starting.
I20260618 22:21:51.807173 19541 tablet_bootstrap.cc:492] T 7573621589a64dd2b458ec974881424f P 2c4af571f140401abccc99931b958257: Bootstrap starting.
I20260618 22:21:51.808790 19539 tablet_bootstrap.cc:654] T 7573621589a64dd2b458ec974881424f P cd629930df894fe4baea4619beea75d5: Neither blocks nor log segments found. Creating new log.
I20260618 22:21:51.809001 19541 tablet_bootstrap.cc:654] T 7573621589a64dd2b458ec974881424f P 2c4af571f140401abccc99931b958257: Neither blocks nor log segments found. Creating new log.
I20260618 22:21:51.809628 19539 log.cc:824] T 7573621589a64dd2b458ec974881424f P cd629930df894fe4baea4619beea75d5: Log is configured to *not* fsync() on all Append() calls
I20260618 22:21:51.809667 19541 log.cc:824] T 7573621589a64dd2b458ec974881424f P 2c4af571f140401abccc99931b958257: Log is configured to *not* fsync() on all Append() calls
I20260618 22:21:51.810264 19540 tablet_bootstrap.cc:492] T 7573621589a64dd2b458ec974881424f P 1160f810d89648b6abea53a588f83fe4: Bootstrap starting.
I20260618 22:21:51.811498 19539 tablet_bootstrap.cc:492] T 7573621589a64dd2b458ec974881424f P cd629930df894fe4baea4619beea75d5: No bootstrap required, opened a new log
I20260618 22:21:51.811596 19541 tablet_bootstrap.cc:492] T 7573621589a64dd2b458ec974881424f P 2c4af571f140401abccc99931b958257: No bootstrap required, opened a new log
I20260618 22:21:51.811725 19539 ts_tablet_manager.cc:1397] T 7573621589a64dd2b458ec974881424f P cd629930df894fe4baea4619beea75d5: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260618 22:21:51.811770 19541 ts_tablet_manager.cc:1397] T 7573621589a64dd2b458ec974881424f P 2c4af571f140401abccc99931b958257: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260618 22:21:51.812301 19540 tablet_bootstrap.cc:654] T 7573621589a64dd2b458ec974881424f P 1160f810d89648b6abea53a588f83fe4: Neither blocks nor log segments found. Creating new log.
I20260618 22:21:51.813067 19540 log.cc:824] T 7573621589a64dd2b458ec974881424f P 1160f810d89648b6abea53a588f83fe4: Log is configured to *not* fsync() on all Append() calls
I20260618 22:21:51.815124 19540 tablet_bootstrap.cc:492] T 7573621589a64dd2b458ec974881424f P 1160f810d89648b6abea53a588f83fe4: No bootstrap required, opened a new log
I20260618 22:21:51.815299 19540 ts_tablet_manager.cc:1397] T 7573621589a64dd2b458ec974881424f P 1160f810d89648b6abea53a588f83fe4: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260618 22:21:51.815675 19539 raft_consensus.cc:348] T 7573621589a64dd2b458ec974881424f P cd629930df894fe4baea4619beea75d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd629930df894fe4baea4619beea75d5" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46365 } } peers { permanent_uuid: "1160f810d89648b6abea53a588f83fe4" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43495 } } peers { permanent_uuid: "2c4af571f140401abccc99931b958257" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33719 } }
I20260618 22:21:51.815923 19539 raft_consensus.cc:374] T 7573621589a64dd2b458ec974881424f P cd629930df894fe4baea4619beea75d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:21:51.815979 19539 raft_consensus.cc:729] T 7573621589a64dd2b458ec974881424f P cd629930df894fe4baea4619beea75d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd629930df894fe4baea4619beea75d5, State: Initialized, Role: FOLLOWER
I20260618 22:21:51.815900 19541 raft_consensus.cc:348] T 7573621589a64dd2b458ec974881424f P 2c4af571f140401abccc99931b958257 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd629930df894fe4baea4619beea75d5" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46365 } } peers { permanent_uuid: "1160f810d89648b6abea53a588f83fe4" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43495 } } peers { permanent_uuid: "2c4af571f140401abccc99931b958257" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33719 } }
I20260618 22:21:51.816145 19541 raft_consensus.cc:374] T 7573621589a64dd2b458ec974881424f P 2c4af571f140401abccc99931b958257 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:21:51.816208 19541 raft_consensus.cc:729] T 7573621589a64dd2b458ec974881424f P 2c4af571f140401abccc99931b958257 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c4af571f140401abccc99931b958257, State: Initialized, Role: FOLLOWER
I20260618 22:21:51.816545 19539 consensus_queue.cc:260] T 7573621589a64dd2b458ec974881424f P cd629930df894fe4baea4619beea75d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd629930df894fe4baea4619beea75d5" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46365 } } peers { permanent_uuid: "1160f810d89648b6abea53a588f83fe4" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43495 } } peers { permanent_uuid: "2c4af571f140401abccc99931b958257" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33719 } }
I20260618 22:21:51.816619 19541 consensus_queue.cc:260] T 7573621589a64dd2b458ec974881424f P 2c4af571f140401abccc99931b958257 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd629930df894fe4baea4619beea75d5" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46365 } } peers { permanent_uuid: "1160f810d89648b6abea53a588f83fe4" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43495 } } peers { permanent_uuid: "2c4af571f140401abccc99931b958257" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33719 } }
I20260618 22:21:51.817211 19260 heartbeater.cc:503] Master 127.0.98.126:46469 was elected leader, sending a full tablet report...
I20260618 22:21:51.817555 19541 ts_tablet_manager.cc:1428] T 7573621589a64dd2b458ec974881424f P 2c4af571f140401abccc99931b958257: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260618 22:21:51.817561 19539 ts_tablet_manager.cc:1428] T 7573621589a64dd2b458ec974881424f P cd629930df894fe4baea4619beea75d5: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260618 22:21:51.817857 19522 heartbeater.cc:503] Master 127.0.98.126:46469 was elected leader, sending a full tablet report...
I20260618 22:21:51.819332 19540 raft_consensus.cc:348] T 7573621589a64dd2b458ec974881424f P 1160f810d89648b6abea53a588f83fe4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd629930df894fe4baea4619beea75d5" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46365 } } peers { permanent_uuid: "1160f810d89648b6abea53a588f83fe4" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43495 } } peers { permanent_uuid: "2c4af571f140401abccc99931b958257" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33719 } }
I20260618 22:21:51.819514 19540 raft_consensus.cc:374] T 7573621589a64dd2b458ec974881424f P 1160f810d89648b6abea53a588f83fe4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:21:51.819545 19540 raft_consensus.cc:729] T 7573621589a64dd2b458ec974881424f P 1160f810d89648b6abea53a588f83fe4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1160f810d89648b6abea53a588f83fe4, State: Initialized, Role: FOLLOWER
I20260618 22:21:51.819991 19540 consensus_queue.cc:260] T 7573621589a64dd2b458ec974881424f P 1160f810d89648b6abea53a588f83fe4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd629930df894fe4baea4619beea75d5" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46365 } } peers { permanent_uuid: "1160f810d89648b6abea53a588f83fe4" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43495 } } peers { permanent_uuid: "2c4af571f140401abccc99931b958257" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33719 } }
I20260618 22:21:51.820971 19540 ts_tablet_manager.cc:1428] T 7573621589a64dd2b458ec974881424f P 1160f810d89648b6abea53a588f83fe4: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260618 22:21:51.821298 19391 heartbeater.cc:503] Master 127.0.98.126:46469 was elected leader, sending a full tablet report...
I20260618 22:21:51.868103 19547 raft_consensus.cc:482] T 7573621589a64dd2b458ec974881424f P 1160f810d89648b6abea53a588f83fe4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:21:51.868364 19547 raft_consensus.cc:504] T 7573621589a64dd2b458ec974881424f P 1160f810d89648b6abea53a588f83fe4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd629930df894fe4baea4619beea75d5" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46365 } } peers { permanent_uuid: "1160f810d89648b6abea53a588f83fe4" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43495 } } peers { permanent_uuid: "2c4af571f140401abccc99931b958257" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33719 } }
W20260618 22:21:51.868996 19261 tablet.cc:2369] T 7573621589a64dd2b458ec974881424f P cd629930df894fe4baea4619beea75d5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:21:51.869810 19547 leader_election.cc:288] T 7573621589a64dd2b458ec974881424f P 1160f810d89648b6abea53a588f83fe4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd629930df894fe4baea4619beea75d5 (127.0.98.65:46365), 2c4af571f140401abccc99931b958257 (127.0.98.67:33719)
I20260618 22:21:51.873546 19538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:51.870209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:50849 (local address 127.0.98.67:33719)
0618 22:21:51.870367 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:51.870371 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:51.870446 (+ 75us) server_negotiation.cc:407] Connection header received
0618 22:21:51.870584 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:51.870586 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:51.870626 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:51.870718 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:21:51.870978 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:51.871498 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:51.872617 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:51.873004 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:51.873124 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:51.873182 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:51.873353 (+ 171us) server_negotiation.cc:299] Negotiation successful
0618 22:21:51.873419 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260618 22:21:51.873574 19393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:51.870091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:33719 (local address 127.0.98.66:50849)
0618 22:21:51.870359 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:51.870376 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0618 22:21:51.870478 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:21:51.870730 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:21:51.870734 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:21:51.870752 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:21:51.870870 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:51.870875 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:51.871618 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:51.871622 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:51.872482 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:51.872491 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:51.873125 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:51.873129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:51.873225 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:51.873252 (+ 27us) client_negotiation.cc:769] Sending connection context
0618 22:21:51.873340 (+ 88us) client_negotiation.cc:240] Negotiation successful
0618 22:21:51.873410 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":50}
I20260618 22:21:51.873608 19534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:51.870113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:58303 (local address 127.0.98.65:46365)
0618 22:21:51.870250 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:51.870257 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:51.870604 (+ 347us) server_negotiation.cc:407] Connection header received
0618 22:21:51.870728 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:51.870731 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:51.870782 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:51.870865 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:21:51.871144 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:51.871635 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:51.872652 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:51.873054 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:51.873170 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:51.873232 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:51.873418 (+ 186us) server_negotiation.cc:299] Negotiation successful
0618 22:21:51.873493 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260618 22:21:51.873626 19548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:51.870110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:46365 (local address 127.0.98.66:58303)
0618 22:21:51.870527 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:51.870542 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:21:51.870620 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:21:51.870876 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:21:51.870879 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:21:51.870894 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:21:51.871029 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:51.871035 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:51.871746 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:51.871752 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:51.872528 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:51.872536 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:51.873174 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:51.873178 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:51.873281 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:51.873309 (+ 28us) client_negotiation.cc:769] Sending connection context
0618 22:21:51.873389 (+ 80us) client_negotiation.cc:240] Negotiation successful
0618 22:21:51.873467 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"mutex_wait_us":3,"thread_start_us":118,"threads_started":1}
I20260618 22:21:51.874322 19476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7573621589a64dd2b458ec974881424f" candidate_uuid: "1160f810d89648b6abea53a588f83fe4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c4af571f140401abccc99931b958257" is_pre_election: true
I20260618 22:21:51.874323 19214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7573621589a64dd2b458ec974881424f" candidate_uuid: "1160f810d89648b6abea53a588f83fe4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd629930df894fe4baea4619beea75d5" is_pre_election: true
I20260618 22:21:51.874627 19476 raft_consensus.cc:2454] T 7573621589a64dd2b458ec974881424f P 2c4af571f140401abccc99931b958257 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1160f810d89648b6abea53a588f83fe4 in term 0.
I20260618 22:21:51.874627 19214 raft_consensus.cc:2454] T 7573621589a64dd2b458ec974881424f P cd629930df894fe4baea4619beea75d5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1160f810d89648b6abea53a588f83fe4 in term 0.
I20260618 22:21:51.875166 19278 leader_election.cc:302] T 7573621589a64dd2b458ec974881424f P 1160f810d89648b6abea53a588f83fe4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1160f810d89648b6abea53a588f83fe4, 2c4af571f140401abccc99931b958257; no voters:
I20260618 22:21:51.875547 19547 raft_consensus.cc:2781] T 7573621589a64dd2b458ec974881424f P 1160f810d89648b6abea53a588f83fe4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:21:51.875686 19547 raft_consensus.cc:482] T 7573621589a64dd2b458ec974881424f P 1160f810d89648b6abea53a588f83fe4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:21:51.875761 19547 raft_consensus.cc:3037] T 7573621589a64dd2b458ec974881424f P 1160f810d89648b6abea53a588f83fe4 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:21:51.876945 19547 raft_consensus.cc:504] T 7573621589a64dd2b458ec974881424f P 1160f810d89648b6abea53a588f83fe4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd629930df894fe4baea4619beea75d5" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46365 } } peers { permanent_uuid: "1160f810d89648b6abea53a588f83fe4" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43495 } } peers { permanent_uuid: "2c4af571f140401abccc99931b958257" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33719 } }
I20260618 22:21:51.877393 19547 leader_election.cc:288] T 7573621589a64dd2b458ec974881424f P 1160f810d89648b6abea53a588f83fe4 [CANDIDATE]: Term 1 election: Requested vote from peers cd629930df894fe4baea4619beea75d5 (127.0.98.65:46365), 2c4af571f140401abccc99931b958257 (127.0.98.67:33719)
I20260618 22:21:51.877725 19214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7573621589a64dd2b458ec974881424f" candidate_uuid: "1160f810d89648b6abea53a588f83fe4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd629930df894fe4baea4619beea75d5"
I20260618 22:21:51.877828 19214 raft_consensus.cc:3037] T 7573621589a64dd2b458ec974881424f P cd629930df894fe4baea4619beea75d5 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:21:51.877813 19476 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7573621589a64dd2b458ec974881424f" candidate_uuid: "1160f810d89648b6abea53a588f83fe4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c4af571f140401abccc99931b958257"
I20260618 22:21:51.878026 19476 raft_consensus.cc:3037] T 7573621589a64dd2b458ec974881424f P 2c4af571f140401abccc99931b958257 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:21:51.879014 19214 raft_consensus.cc:2454] T 7573621589a64dd2b458ec974881424f P cd629930df894fe4baea4619beea75d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1160f810d89648b6abea53a588f83fe4 in term 1.
I20260618 22:21:51.879244 19476 raft_consensus.cc:2454] T 7573621589a64dd2b458ec974881424f P 2c4af571f140401abccc99931b958257 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1160f810d89648b6abea53a588f83fe4 in term 1.
I20260618 22:21:51.879387 19280 leader_election.cc:302] T 7573621589a64dd2b458ec974881424f P 1160f810d89648b6abea53a588f83fe4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1160f810d89648b6abea53a588f83fe4, cd629930df894fe4baea4619beea75d5; no voters:
I20260618 22:21:51.879642 19547 raft_consensus.cc:2781] T 7573621589a64dd2b458ec974881424f P 1160f810d89648b6abea53a588f83fe4 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:21:51.879909 19547 raft_consensus.cc:686] T 7573621589a64dd2b458ec974881424f P 1160f810d89648b6abea53a588f83fe4 [term 1 LEADER]: Becoming Leader. State: Replica: 1160f810d89648b6abea53a588f83fe4, State: Running, Role: LEADER
I20260618 22:21:51.880261 19547 consensus_queue.cc:237] T 7573621589a64dd2b458ec974881424f P 1160f810d89648b6abea53a588f83fe4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd629930df894fe4baea4619beea75d5" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46365 } } peers { permanent_uuid: "1160f810d89648b6abea53a588f83fe4" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43495 } } peers { permanent_uuid: "2c4af571f140401abccc99931b958257" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33719 } }
I20260618 22:21:51.883741 19071 catalog_manager.cc:5495] T 7573621589a64dd2b458ec974881424f P 1160f810d89648b6abea53a588f83fe4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1160f810d89648b6abea53a588f83fe4 (127.0.98.66). New cstate: current_term: 1 leader_uuid: "1160f810d89648b6abea53a588f83fe4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd629930df894fe4baea4619beea75d5" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46365 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1160f810d89648b6abea53a588f83fe4" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43495 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2c4af571f140401abccc99931b958257" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33719 } health_report { overall_health: UNKNOWN } } }
I20260618 22:21:51.895387 19537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:51.891943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51794 (local address 127.0.98.66:43495)
0618 22:21:51.892058 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:51.892062 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:51.892073 (+ 11us) server_negotiation.cc:407] Connection header received
0618 22:21:51.892129 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:51.892131 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:51.892179 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:51.892265 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:21:51.892576 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:51.893081 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:51.894024 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:51.894240 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:51.894324 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:51.894570 (+ 246us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:21:51.894964 (+ 394us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:21:51.895076 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:51.895235 (+ 159us) server_negotiation.cc:299] Negotiation successful
0618 22:21:51.895266 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260618 22:21:51.901388 19476 raft_consensus.cc:1261] T 7573621589a64dd2b458ec974881424f P 2c4af571f140401abccc99931b958257 [term 1 FOLLOWER]: Refusing update from remote peer 1160f810d89648b6abea53a588f83fe4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:21:51.901464 19214 raft_consensus.cc:1261] T 7573621589a64dd2b458ec974881424f P cd629930df894fe4baea4619beea75d5 [term 1 FOLLOWER]: Refusing update from remote peer 1160f810d89648b6abea53a588f83fe4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:21:51.902159 19549 consensus_queue.cc:1035] T 7573621589a64dd2b458ec974881424f P 1160f810d89648b6abea53a588f83fe4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c4af571f140401abccc99931b958257" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 33719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:21:51.902364 19547 consensus_queue.cc:1035] T 7573621589a64dd2b458ec974881424f P 1160f810d89648b6abea53a588f83fe4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd629930df894fe4baea4619beea75d5" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 46365 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:21:51.909396 19553 mvcc.cc:205] Tried to move back new op lower bound from 7298340093540929536 to 7298340093464649728. Current Snapshot: MvccSnapshot[applied={T|T < 7298340093540929536}]
I20260618 22:21:51.918341 19070 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:58088:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:21:51.918515 19070 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:58088:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:21:51.920845 19070 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 486161f1d2a74f808c2590417ea08793: Sending DeleteTablet for 3 replicas of tablet 7573621589a64dd2b458ec974881424f
I20260618 22:21:51.921685 19194 tablet_service.cc:1514] Processing DeleteTablet for tablet 7573621589a64dd2b458ec974881424f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:21:51 UTC) from {username='slave'} at 127.0.0.1:60846
I20260618 22:21:51.921664 19325 tablet_service.cc:1514] Processing DeleteTablet for tablet 7573621589a64dd2b458ec974881424f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:21:51 UTC) from {username='slave'} at 127.0.0.1:51778
I20260618 22:21:51.921850 19456 tablet_service.cc:1514] Processing DeleteTablet for tablet 7573621589a64dd2b458ec974881424f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:21:51 UTC) from {username='slave'} at 127.0.0.1:40852
I20260618 22:21:51.922188 19562 tablet_replica.cc:330] T 7573621589a64dd2b458ec974881424f P 1160f810d89648b6abea53a588f83fe4: stopping tablet replica
I20260618 22:21:51.922358 19564 tablet_replica.cc:330] T 7573621589a64dd2b458ec974881424f P 2c4af571f140401abccc99931b958257: stopping tablet replica
I20260618 22:21:51.922504 19562 raft_consensus.cc:2229] T 7573621589a64dd2b458ec974881424f P 1160f810d89648b6abea53a588f83fe4 [term 1 LEADER]: Raft consensus shutting down.
I20260618 22:21:51.922590 19564 raft_consensus.cc:2229] T 7573621589a64dd2b458ec974881424f P 2c4af571f140401abccc99931b958257 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:21:51.922864 19563 tablet_replica.cc:330] T 7573621589a64dd2b458ec974881424f P cd629930df894fe4baea4619beea75d5: stopping tablet replica
I20260618 22:21:51.922994 19564 raft_consensus.cc:2258] T 7573621589a64dd2b458ec974881424f P 2c4af571f140401abccc99931b958257 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:21:51.923207 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 19132
I20260618 22:21:51.923331 19563 raft_consensus.cc:2229] T 7573621589a64dd2b458ec974881424f P cd629930df894fe4baea4619beea75d5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:21:51.925055 19564 ts_tablet_manager.cc:1905] T 7573621589a64dd2b458ec974881424f P 2c4af571f140401abccc99931b958257: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:21:51.926470 19562 raft_consensus.cc:2258] T 7573621589a64dd2b458ec974881424f P 1160f810d89648b6abea53a588f83fe4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:21:51.927551 19564 ts_tablet_manager.cc:1918] T 7573621589a64dd2b458ec974881424f P 2c4af571f140401abccc99931b958257: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:21:51.927664 19564 log.cc:1196] T 7573621589a64dd2b458ec974881424f P 2c4af571f140401abccc99931b958257: Deleting WAL directory at /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781821065380026-393-0/minicluster-data/ts-2/wal/wals/7573621589a64dd2b458ec974881424f
I20260618 22:21:51.927942 19564 ts_tablet_manager.cc:1939] T 7573621589a64dd2b458ec974881424f P 2c4af571f140401abccc99931b958257: Deleting consensus metadata
I20260618 22:21:51.929109 19056 catalog_manager.cc:4841] TS 2c4af571f140401abccc99931b958257 (127.0.98.67:33719): tablet 7573621589a64dd2b458ec974881424f (table test-table [id=8081a01e9e5a445d8882f3f1e2816bc6]) successfully deleted
W20260618 22:21:51.931187 19058 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.65:46365 (error 108)
I20260618 22:21:51.931383 19562 ts_tablet_manager.cc:1905] T 7573621589a64dd2b458ec974881424f P 1160f810d89648b6abea53a588f83fe4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:21:51.931980 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 19263
I20260618 22:21:51.933322 19536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:51.932338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:46365 (local address 127.0.0.1:60854)
0618 22:21:51.933160 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:51.933239 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.65:46365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":718}
W20260618 22:21:51.933511 19058 catalog_manager.cc:4584] TS cd629930df894fe4baea4619beea75d5 (127.0.98.65:46365): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7573621589a64dd2b458ec974881424f: Network error: Client connection negotiation failed: client connection to 127.0.98.65:46365: connect: Connection refused (error 111)
I20260618 22:21:51.937711 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 19394
I20260618 22:21:51.939535 19536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:51.937959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:43495 (local address 127.0.0.1:51800)
0618 22:21:51.939419 (+ 1460us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:51.939467 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.66:43495: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1382}
I20260618 22:21:51.942790 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 19041
2026-06-18T22:21:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1027 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:21:52 dist-test-slave-g6r4 krb5kdc[19568](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:21:52 dist-test-slave-g6r4 krb5kdc[19568](info): set up 2 sockets
Jun 18 22:21:52 dist-test-slave-g6r4 krb5kdc[19568](info): commencing operation
krb5kdc: starting...
W20260618 22:21:52.989961 393 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/krb5kdc/test-user.keytab.
Jun 18 22:21:53 dist-test-slave-g6r4 krb5kdc[19568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:21:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:21:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260618 22:21:53.109711 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:34719
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:34043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:34719
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:21:53.216836 19581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:21:53.217108 19581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:21:53.217151 19581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:21:53.221284 19581 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:21:53.221336 19581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:21:53.221355 19581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:21:53.221371 19581 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:21:53.221386 19581 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:21:53.226408 19581 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:34043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:34719
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:34719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:21:53.227304 19581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:21:53.228276 19581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:21:53.236565 19589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:21:53.236577 19586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:21:53.236575 19587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:21:53.236733 19581 server_base.cc:1034] running on GCE node
I20260618 22:21:53.237368 19581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:21:53.238446 19581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:21:53.239601 19581 hybrid_clock.cc:648] HybridClock initialized: now 1781821313239585 us; error 31 us; skew 500 ppm
Jun 18 22:21:53 dist-test-slave-g6r4 krb5kdc[19568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821313, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:21:53.242326 19581 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260618 22:21:53.243222 19581 webserver.cc:460] Webserver started at http://127.0.98.126:36075/ using document root <none> and password file <none>
I20260618 22:21:53.243737 19581 fs_manager.cc:359] Metadata directory not provided
I20260618 22:21:53.243796 19581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:21:53.244040 19581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:21:53.246038 19581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/minicluster-data/master-0/data/instance:
uuid: "d37a62d43de64edcb13dc3155ac51628"
format_stamp: "Formatted at 2026-06-18 22:21:53 on dist-test-slave-g6r4"
server_key: "13b54f1deba8db96c9e8fa8fa7a4d08b"
server_key_iv: "9e95aa0180f1c4efa725bdfbcfdcf990"
server_key_version: "encryptionkey@0"
I20260618 22:21:53.246584 19581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/minicluster-data/master-0/wal/instance:
uuid: "d37a62d43de64edcb13dc3155ac51628"
format_stamp: "Formatted at 2026-06-18 22:21:53 on dist-test-slave-g6r4"
server_key: "13b54f1deba8db96c9e8fa8fa7a4d08b"
server_key_iv: "9e95aa0180f1c4efa725bdfbcfdcf990"
server_key_version: "encryptionkey@0"
I20260618 22:21:53.250088 19581 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:21:53.252506 19596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:53.253696 19581 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:21:53.253829 19581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/minicluster-data/master-0/wal
uuid: "d37a62d43de64edcb13dc3155ac51628"
format_stamp: "Formatted at 2026-06-18 22:21:53 on dist-test-slave-g6r4"
server_key: "13b54f1deba8db96c9e8fa8fa7a4d08b"
server_key_iv: "9e95aa0180f1c4efa725bdfbcfdcf990"
server_key_version: "encryptionkey@0"
I20260618 22:21:53.254017 19581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:21:53.268698 19581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:21:53.271777 19581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:21:53.271966 19581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:21:53.278795 19581 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:34719
I20260618 22:21:53.278807 19648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:34719 every 8 connection(s)
I20260618 22:21:53.279726 19581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
I20260618 22:21:53.282531 19649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:21:53.285004 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 19581
I20260618 22:21:53.285167 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/minicluster-data/master-0/wal/instance
I20260618 22:21:53.285463 393 external_mini_cluster.cc:1428] Setting key 399f6537c182f1bce3c2d0a58d8efaa1
I20260618 22:21:53.288231 19649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d37a62d43de64edcb13dc3155ac51628: Bootstrap starting.
I20260618 22:21:53.290407 19649 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d37a62d43de64edcb13dc3155ac51628: Neither blocks nor log segments found. Creating new log.
I20260618 22:21:53.291114 19649 log.cc:824] T 00000000000000000000000000000000 P d37a62d43de64edcb13dc3155ac51628: Log is configured to *not* fsync() on all Append() calls
I20260618 22:21:53.292932 19649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d37a62d43de64edcb13dc3155ac51628: No bootstrap required, opened a new log
Jun 18 22:21:53 dist-test-slave-g6r4 krb5kdc[19568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:21:53.296543 19649 raft_consensus.cc:348] T 00000000000000000000000000000000 P d37a62d43de64edcb13dc3155ac51628 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d37a62d43de64edcb13dc3155ac51628" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34719 } }
I20260618 22:21:53.296804 19649 raft_consensus.cc:374] T 00000000000000000000000000000000 P d37a62d43de64edcb13dc3155ac51628 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:21:53.296885 19649 raft_consensus.cc:729] T 00000000000000000000000000000000 P d37a62d43de64edcb13dc3155ac51628 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d37a62d43de64edcb13dc3155ac51628, State: Initialized, Role: FOLLOWER
I20260618 22:21:53.297317 19649 consensus_queue.cc:260] T 00000000000000000000000000000000 P d37a62d43de64edcb13dc3155ac51628 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d37a62d43de64edcb13dc3155ac51628" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34719 } }
I20260618 22:21:53.297477 19649 raft_consensus.cc:388] T 00000000000000000000000000000000 P d37a62d43de64edcb13dc3155ac51628 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:21:53.297572 19649 raft_consensus.cc:482] T 00000000000000000000000000000000 P d37a62d43de64edcb13dc3155ac51628 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:21:53.297676 19649 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d37a62d43de64edcb13dc3155ac51628 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:21:53.298869 19649 raft_consensus.cc:504] T 00000000000000000000000000000000 P d37a62d43de64edcb13dc3155ac51628 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d37a62d43de64edcb13dc3155ac51628" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34719 } }
I20260618 22:21:53.299219 19649 leader_election.cc:302] T 00000000000000000000000000000000 P d37a62d43de64edcb13dc3155ac51628 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d37a62d43de64edcb13dc3155ac51628; no voters:
I20260618 22:21:53.299554 19649 leader_election.cc:288] T 00000000000000000000000000000000 P d37a62d43de64edcb13dc3155ac51628 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:21:53.299664 19654 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d37a62d43de64edcb13dc3155ac51628 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:21:53.299887 19652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:53.286796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44598 (local address 127.0.98.126:34719)
0618 22:21:53.287186 (+ 390us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:53.287201 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:53.287235 (+ 34us) server_negotiation.cc:407] Connection header received
0618 22:21:53.287817 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:53.287832 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:53.288159 (+ 327us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:53.288488 (+ 329us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:21:53.289271 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:53.290324 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:53.291219 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:53.291715 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:53.291889 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:53.294394 (+ 2505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:21:53.294400 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:21:53.294412 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:21:53.294437 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:21:53.296530 (+ 2093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:21:53.297066 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:21:53.297072 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:21:53.297076 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:21:53.297234 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:21:53.297566 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:21:53.297570 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:21:53.297572 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:21:53.298127 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:21:53.298333 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:53.298600 (+ 267us) server_negotiation.cc:299] Negotiation successful
0618 22:21:53.298907 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":133,"threads_started":1}
I20260618 22:21:53.300633 19649 sys_catalog.cc:564] T 00000000000000000000000000000000 P d37a62d43de64edcb13dc3155ac51628 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:21:53.299990 19654 raft_consensus.cc:686] T 00000000000000000000000000000000 P d37a62d43de64edcb13dc3155ac51628 [term 1 LEADER]: Becoming Leader. State: Replica: d37a62d43de64edcb13dc3155ac51628, State: Running, Role: LEADER
I20260618 22:21:53.301256 19654 consensus_queue.cc:237] T 00000000000000000000000000000000 P d37a62d43de64edcb13dc3155ac51628 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d37a62d43de64edcb13dc3155ac51628" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34719 } }
I20260618 22:21:53.303228 19655 sys_catalog.cc:455] T 00000000000000000000000000000000 P d37a62d43de64edcb13dc3155ac51628 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d37a62d43de64edcb13dc3155ac51628. Latest consensus state: current_term: 1 leader_uuid: "d37a62d43de64edcb13dc3155ac51628" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d37a62d43de64edcb13dc3155ac51628" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34719 } } }
I20260618 22:21:53.303342 19655 sys_catalog.cc:458] T 00000000000000000000000000000000 P d37a62d43de64edcb13dc3155ac51628 [sys.catalog]: This master's current role is: LEADER
I20260618 22:21:53.303588 19656 sys_catalog.cc:455] T 00000000000000000000000000000000 P d37a62d43de64edcb13dc3155ac51628 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d37a62d43de64edcb13dc3155ac51628" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d37a62d43de64edcb13dc3155ac51628" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34719 } } }
I20260618 22:21:53.303686 19656 sys_catalog.cc:458] T 00000000000000000000000000000000 P d37a62d43de64edcb13dc3155ac51628 [sys.catalog]: This master's current role is: LEADER
W20260618 22:21:53.305472 19668 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d37a62d43de64edcb13dc3155ac51628: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 22:21:53.305536 19668 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 22:21:53.305615 19669 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:21:53.306360 19669 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:21:53.311647 19669 catalog_manager.cc:1347] Generated new cluster ID: 19f719bf19eb4f99b307d394f98828d9
I20260618 22:21:53.311708 19669 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:21:53.321602 19669 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:21:53.322732 19669 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:21:53.340791 19669 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d37a62d43de64edcb13dc3155ac51628: Generated new TSK 0
I20260618 22:21:53.341548 19669 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab.
Entry for principal kudu/127.0.98.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab.
I20260618 22:21:53.378990 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:34719
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:34043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:21:53.487433 19675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:21:53.487663 19675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:21:53.487705 19675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:21:53.491521 19675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:21:53.491564 19675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:21:53.491619 19675 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260618 22:21:53.496564 19675 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:34043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/krb5kdc/kudu_127.0.98.65.keytab
--principal=kudu/127.0.98.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:34719
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19675
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:21:53.497396 19675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:21:53.498282 19675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:21:53.507650 19681 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:21:53.507647 19680 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:21:53.507679 19683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:21:53.507946 19675 server_base.cc:1034] running on GCE node
I20260618 22:21:53.508306 19675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:21:53.508956 19675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:21:53.510147 19675 hybrid_clock.cc:648] HybridClock initialized: now 1781821313510113 us; error 43 us; skew 500 ppm
Jun 18 22:21:53 dist-test-slave-g6r4 krb5kdc[19568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821313, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:21:53.512960 19675 init.cc:377] Logged in from keytab as kudu/127.0.98.65@KRBTEST.COM (short username kudu)
I20260618 22:21:53.513823 19675 webserver.cc:460] Webserver started at http://127.0.98.65:41397/ using document root <none> and password file <none>
I20260618 22:21:53.514500 19675 fs_manager.cc:359] Metadata directory not provided
I20260618 22:21:53.514600 19675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:21:53.514832 19675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:21:53.516897 19675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/minicluster-data/ts-0/data/instance:
uuid: "717954a6fb284a85b2599f4705fafe42"
format_stamp: "Formatted at 2026-06-18 22:21:53 on dist-test-slave-g6r4"
server_key: "3aec22a00672213de13cd3806c2ba559"
server_key_iv: "0a21bbef29cd1477db4084254199cae7"
server_key_version: "encryptionkey@0"
I20260618 22:21:53.517490 19675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/minicluster-data/ts-0/wal/instance:
uuid: "717954a6fb284a85b2599f4705fafe42"
format_stamp: "Formatted at 2026-06-18 22:21:53 on dist-test-slave-g6r4"
server_key: "3aec22a00672213de13cd3806c2ba559"
server_key_iv: "0a21bbef29cd1477db4084254199cae7"
server_key_version: "encryptionkey@0"
I20260618 22:21:53.521255 19675 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260618 22:21:53.523903 19690 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:53.525027 19675 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260618 22:21:53.525156 19675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/minicluster-data/ts-0/wal
uuid: "717954a6fb284a85b2599f4705fafe42"
format_stamp: "Formatted at 2026-06-18 22:21:53 on dist-test-slave-g6r4"
server_key: "3aec22a00672213de13cd3806c2ba559"
server_key_iv: "0a21bbef29cd1477db4084254199cae7"
server_key_version: "encryptionkey@0"
I20260618 22:21:53.525256 19675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:21:53.551882 19675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:21:53.555079 19675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:21:53.555272 19675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:21:53.555833 19675 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:21:53.556704 19675 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:21:53.556774 19675 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:53.556844 19675 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:21:53.556895 19675 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:53.565659 19675 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:34657
I20260618 22:21:53.565686 19803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:34657 every 8 connection(s)
I20260618 22:21:53.566537 19675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
Jun 18 22:21:53 dist-test-slave-g6r4 krb5kdc[19568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821313, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.65@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:21:53.575037 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 19675
I20260618 22:21:53.575201 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/minicluster-data/ts-0/wal/instance
I20260618 22:21:53.575593 393 external_mini_cluster.cc:1428] Setting key 10c6088a2c580b17cb16f9aa46018f73
I20260618 22:21:53.579391 19652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:53.568209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:49271 (local address 127.0.98.126:34719)
0618 22:21:53.568342 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:53.568346 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:53.569196 (+ 850us) server_negotiation.cc:407] Connection header received
0618 22:21:53.569998 (+ 802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:53.570001 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:53.570044 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:53.570114 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:21:53.571173 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:53.571707 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:53.572773 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:53.573024 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:53.573104 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:53.575475 (+ 2371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:21:53.575480 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:21:53.575482 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:21:53.575512 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:21:53.577026 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:21:53.577514 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:21:53.577520 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:21:53.577525 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:21:53.577610 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:21:53.578095 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:21:53.578101 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:21:53.578106 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:21:53.578354 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:21:53.578459 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:53.579038 (+ 579us) server_negotiation.cc:299] Negotiation successful
0618 22:21:53.579175 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260618 22:21:53.579965 19806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:53.568627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34719 (local address 127.0.98.65:49271)
0618 22:21:53.569038 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:53.569088 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0618 22:21:53.569816 (+ 728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:21:53.570238 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:21:53.570244 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:21:53.570600 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:21:53.571037 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:53.571052 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:53.571815 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:53.571818 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:53.572663 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:53.572672 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:53.573146 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:53.573148 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:53.573274 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:53.573855 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:21:53.573869 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:21:53.575310 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:21:53.577154 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:21:53.577158 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:21:53.577168 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:21:53.577410 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:21:53.577711 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:21:53.577713 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:21:53.577715 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:21:53.578001 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:21:53.578455 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:21:53.578460 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:21:53.578697 (+ 237us) client_negotiation.cc:769] Sending connection context
0618 22:21:53.578942 (+ 245us) client_negotiation.cc:240] Negotiation successful
0618 22:21:53.579175 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":108,"threads_started":1}
I20260618 22:21:53.581082 19804 heartbeater.cc:347] Connected to a master server at 127.0.98.126:34719
I20260618 22:21:53.581307 19804 heartbeater.cc:464] Registering TS with master...
I20260618 22:21:53.581743 19804 heartbeater.cc:511] Master 127.0.98.126:34719 requested a full tablet report, sending...
I20260618 22:21:53.583334 19613 ts_manager.cc:194] Registered new tserver with Master: 717954a6fb284a85b2599f4705fafe42 (127.0.98.65:34657)
I20260618 22:21:53.584946 19613 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.65@KRBTEST.COM'} at 127.0.98.65:49271
WARNING: no policy specified for kudu/127.0.98.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab.
Entry for principal kudu/127.0.98.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab.
I20260618 22:21:53.604632 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:34719
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:34043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:21:53.712828 19809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:21:53.713059 19809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:21:53.713099 19809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:21:53.718426 19809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:21:53.718480 19809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:21:53.718564 19809 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260618 22:21:53.723608 19809 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:34043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/krb5kdc/kudu_127.0.98.66.keytab
--principal=kudu/127.0.98.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:34719
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:21:53.724535 19809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:21:53.725478 19809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:21:53.734715 19817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:21:53.734673 19814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:21:53.735016 19809 server_base.cc:1034] running on GCE node
W20260618 22:21:53.734768 19815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:21:53.735391 19809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:21:53.736020 19809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:21:53.737207 19809 hybrid_clock.cc:648] HybridClock initialized: now 1781821313737192 us; error 26 us; skew 500 ppm
Jun 18 22:21:53 dist-test-slave-g6r4 krb5kdc[19568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821313, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:21:53.740011 19809 init.cc:377] Logged in from keytab as kudu/127.0.98.66@KRBTEST.COM (short username kudu)
I20260618 22:21:53.740726 19809 webserver.cc:460] Webserver started at http://127.0.98.66:38757/ using document root <none> and password file <none>
I20260618 22:21:53.741205 19809 fs_manager.cc:359] Metadata directory not provided
I20260618 22:21:53.741258 19809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:21:53.741407 19809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:21:53.743299 19809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/minicluster-data/ts-1/data/instance:
uuid: "310921fdc75642cdb086aef7a2c75980"
format_stamp: "Formatted at 2026-06-18 22:21:53 on dist-test-slave-g6r4"
server_key: "889f16d95e6d60dfdaf567bca7cbfa9b"
server_key_iv: "9db2302a34dca7df333a57873b3616dd"
server_key_version: "encryptionkey@0"
I20260618 22:21:53.743800 19809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/minicluster-data/ts-1/wal/instance:
uuid: "310921fdc75642cdb086aef7a2c75980"
format_stamp: "Formatted at 2026-06-18 22:21:53 on dist-test-slave-g6r4"
server_key: "889f16d95e6d60dfdaf567bca7cbfa9b"
server_key_iv: "9db2302a34dca7df333a57873b3616dd"
server_key_version: "encryptionkey@0"
I20260618 22:21:53.747259 19809 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:21:53.749465 19824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:53.750873 19809 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:21:53.750967 19809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/minicluster-data/ts-1/wal
uuid: "310921fdc75642cdb086aef7a2c75980"
format_stamp: "Formatted at 2026-06-18 22:21:53 on dist-test-slave-g6r4"
server_key: "889f16d95e6d60dfdaf567bca7cbfa9b"
server_key_iv: "9db2302a34dca7df333a57873b3616dd"
server_key_version: "encryptionkey@0"
I20260618 22:21:53.751089 19809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:21:53.779706 19809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:21:53.782619 19809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:21:53.782820 19809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:21:53.783427 19809 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:21:53.784199 19809 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:21:53.784242 19809 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:53.784302 19809 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:21:53.784336 19809 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:53.793123 19809 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:43787
I20260618 22:21:53.793161 19937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:43787 every 8 connection(s)
I20260618 22:21:53.793964 19809 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
I20260618 22:21:53.800657 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 19809
I20260618 22:21:53.800810 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/minicluster-data/ts-1/wal/instance
I20260618 22:21:53.801146 393 external_mini_cluster.cc:1428] Setting key a2b53cf374474af5f0df4d968de1d0b1
Jun 18 22:21:53 dist-test-slave-g6r4 krb5kdc[19568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821313, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.66@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:21:53.808382 19652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:53.795618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:55823 (local address 127.0.98.126:34719)
0618 22:21:53.795756 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:53.795761 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:53.796818 (+ 1057us) server_negotiation.cc:407] Connection header received
0618 22:21:53.797642 (+ 824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:53.797645 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:53.797688 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:53.797776 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:21:53.799126 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:53.799658 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:53.800947 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:53.801174 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:53.801288 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:53.804351 (+ 3063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:21:53.804356 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:21:53.804360 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:21:53.804389 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:21:53.806098 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:21:53.806664 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:21:53.806671 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:21:53.806674 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:21:53.806769 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:21:53.807148 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:21:53.807153 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:21:53.807156 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:21:53.807404 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:21:53.807527 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:53.808005 (+ 478us) server_negotiation.cc:299] Negotiation successful
0618 22:21:53.808147 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260618 22:21:53.808985 19940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:53.796043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34719 (local address 127.0.98.66:55823)
0618 22:21:53.796680 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:53.796720 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0618 22:21:53.797456 (+ 736us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:21:53.797957 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:21:53.797965 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:21:53.798495 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:21:53.798964 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:53.798986 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:53.799786 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:53.799790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:53.800818 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:53.800829 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:53.801366 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:53.801371 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:53.801595 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:53.802242 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:21:53.802258 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:21:53.804173 (+ 1915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:21:53.806229 (+ 2056us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:21:53.806234 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:21:53.806246 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:21:53.806545 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:21:53.806885 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:21:53.806888 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:21:53.806890 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:21:53.807041 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:21:53.807532 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:21:53.807537 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:21:53.807745 (+ 208us) client_negotiation.cc:769] Sending connection context
0618 22:21:53.807924 (+ 179us) client_negotiation.cc:240] Negotiation successful
0618 22:21:53.808158 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":107,"threads_started":1}
I20260618 22:21:53.810220 19938 heartbeater.cc:347] Connected to a master server at 127.0.98.126:34719
I20260618 22:21:53.810469 19938 heartbeater.cc:464] Registering TS with master...
I20260618 22:21:53.811025 19938 heartbeater.cc:511] Master 127.0.98.126:34719 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.67@KRBTEST.COM; defaulting to no policy
I20260618 22:21:53.812079 19613 ts_manager.cc:194] Registered new tserver with Master: 310921fdc75642cdb086aef7a2c75980 (127.0.98.66:43787)
I20260618 22:21:53.812660 19613 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.66@KRBTEST.COM'} at 127.0.98.66:55823
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab.
Entry for principal kudu/127.0.98.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab.
I20260618 22:21:53.828716 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:34719
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:34043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:21:53.941388 19943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:21:53.941632 19943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:21:53.941675 19943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:21:53.945640 19943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:21:53.945693 19943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:21:53.945761 19943 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260618 22:21:53.951133 19943 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:34043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/krb5kdc/kudu_127.0.98.67.keytab
--principal=kudu/127.0.98.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:34719
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:21:53.952088 19943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:21:53.953063 19943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:21:53.962231 19948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:21:53.962217 19949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:21:53.962216 19951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:21:53.962420 19943 server_base.cc:1034] running on GCE node
I20260618 22:21:53.962913 19943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:21:53.963508 19943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:21:53.964670 19943 hybrid_clock.cc:648] HybridClock initialized: now 1781821313964659 us; error 24 us; skew 500 ppm
Jun 18 22:21:53 dist-test-slave-g6r4 krb5kdc[19568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821313, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:21:53.967311 19943 init.cc:377] Logged in from keytab as kudu/127.0.98.67@KRBTEST.COM (short username kudu)
I20260618 22:21:53.968106 19943 webserver.cc:460] Webserver started at http://127.0.98.67:36393/ using document root <none> and password file <none>
I20260618 22:21:53.968616 19943 fs_manager.cc:359] Metadata directory not provided
I20260618 22:21:53.968674 19943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:21:53.968866 19943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:21:53.970837 19943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/minicluster-data/ts-2/data/instance:
uuid: "4f4388a49b274f47be007b57c076597e"
format_stamp: "Formatted at 2026-06-18 22:21:53 on dist-test-slave-g6r4"
server_key: "32fe1a572fbf233a981068a371106403"
server_key_iv: "083056a2da435a44fc3a097c49c4b74c"
server_key_version: "encryptionkey@0"
I20260618 22:21:53.971369 19943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/minicluster-data/ts-2/wal/instance:
uuid: "4f4388a49b274f47be007b57c076597e"
format_stamp: "Formatted at 2026-06-18 22:21:53 on dist-test-slave-g6r4"
server_key: "32fe1a572fbf233a981068a371106403"
server_key_iv: "083056a2da435a44fc3a097c49c4b74c"
server_key_version: "encryptionkey@0"
I20260618 22:21:53.974800 19943 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:21:53.977000 19958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:53.978243 19943 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:21:53.978367 19943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/minicluster-data/ts-2/wal
uuid: "4f4388a49b274f47be007b57c076597e"
format_stamp: "Formatted at 2026-06-18 22:21:53 on dist-test-slave-g6r4"
server_key: "32fe1a572fbf233a981068a371106403"
server_key_iv: "083056a2da435a44fc3a097c49c4b74c"
server_key_version: "encryptionkey@0"
I20260618 22:21:53.978482 19943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:21:53.998649 19943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:21:54.001641 19943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:21:54.001832 19943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:21:54.002483 19943 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:21:54.003261 19943 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:21:54.003330 19943 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:54.003398 19943 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:21:54.003445 19943 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:54.012076 19943 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:46305
I20260618 22:21:54.012108 20071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:46305 every 8 connection(s)
I20260618 22:21:54.012902 19943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
I20260618 22:21:54.014554 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 19943
I20260618 22:21:54.014699 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/minicluster-data/ts-2/wal/instance
I20260618 22:21:54.014982 393 external_mini_cluster.cc:1428] Setting key 18d4307d05950910b23a42895b3a4e29
Jun 18 22:21:54 dist-test-slave-g6r4 krb5kdc[19568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821313, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.67@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:21:54.025745 19652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:54.014620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:47267 (local address 127.0.98.126:34719)
0618 22:21:54.014774 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:54.014778 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:54.015669 (+ 891us) server_negotiation.cc:407] Connection header received
0618 22:21:54.016785 (+ 1116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:54.016788 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:54.016839 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:54.016929 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:21:54.018126 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:54.018665 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:54.019760 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:54.019982 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:54.020069 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:54.022399 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:21:54.022402 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:21:54.022404 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:21:54.022427 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:21:54.023799 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:21:54.024291 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:21:54.024295 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:21:54.024297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:21:54.024379 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:21:54.024726 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:21:54.024731 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:21:54.024733 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:21:54.024909 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:21:54.025031 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:54.025466 (+ 435us) server_negotiation.cc:299] Negotiation successful
0618 22:21:54.025582 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260618 22:21:54.026556 20074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:54.015010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:34719 (local address 127.0.98.67:47267)
0618 22:21:54.015525 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:54.015579 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0618 22:21:54.016591 (+ 1012us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:21:54.017129 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:21:54.017136 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:21:54.017528 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:21:54.018000 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:54.018013 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:54.018764 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:54.018766 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:54.019625 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:54.019633 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:54.020105 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:54.020108 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:54.020237 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:54.020721 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:21:54.020736 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:21:54.022243 (+ 1507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:21:54.023932 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:21:54.023937 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:21:54.023957 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:21:54.024187 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:21:54.024499 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:21:54.024503 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:21:54.024506 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:21:54.024630 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:21:54.025031 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:21:54.025037 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:21:54.025248 (+ 211us) client_negotiation.cc:769] Sending connection context
0618 22:21:54.025423 (+ 175us) client_negotiation.cc:240] Negotiation successful
0618 22:21:54.025714 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":152,"threads_started":1}
I20260618 22:21:54.027664 20072 heartbeater.cc:347] Connected to a master server at 127.0.98.126:34719
I20260618 22:21:54.027966 20072 heartbeater.cc:464] Registering TS with master...
I20260618 22:21:54.028529 20072 heartbeater.cc:511] Master 127.0.98.126:34719 requested a full tablet report, sending...
I20260618 22:21:54.029459 19613 ts_manager.cc:194] Registered new tserver with Master: 4f4388a49b274f47be007b57c076597e (127.0.98.67:46305)
I20260618 22:21:54.030205 19613 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.67@KRBTEST.COM'} at 127.0.98.67:47267
I20260618 22:21:54.040234 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 22:21:54.049322 19652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:54.042148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33628 (local address 127.0.98.126:34719)
0618 22:21:54.042268 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:54.042271 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:54.042446 (+ 175us) server_negotiation.cc:407] Connection header received
0618 22:21:54.042544 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:54.042546 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:54.042583 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:54.042650 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:21:54.043254 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:54.043753 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:54.044628 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:54.044844 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:54.044938 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:54.046540 (+ 1602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:21:54.046543 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:21:54.046545 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:21:54.046566 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:21:54.047808 (+ 1242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:21:54.048235 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:21:54.048238 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:21:54.048239 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:21:54.048313 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:21:54.048557 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:21:54.048560 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:21:54.048561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:21:54.048766 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:21:54.048868 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:54.049120 (+ 252us) server_negotiation.cc:299] Negotiation successful
0618 22:21:54.049209 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260618 22:21:54.052347 19613 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33628:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 22:21:54.055490 19613 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:21:54.069614 20084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:54.063458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55176 (local address 127.0.98.65:34657)
0618 22:21:54.063975 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:54.063993 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:54.064017 (+ 24us) server_negotiation.cc:407] Connection header received
0618 22:21:54.064239 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:54.064247 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:54.064511 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:54.065037 (+ 526us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:21:54.065187 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:54.066315 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:54.068234 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:54.068994 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:54.069086 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:54.069195 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:54.069347 (+ 152us) server_negotiation.cc:299] Negotiation successful
0618 22:21:54.069440 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":69,"threads_started":1}
I20260618 22:21:54.069630 20085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:54.063607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:46305 (local address 127.0.0.1:59204)
0618 22:21:54.064219 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:54.064232 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 22:21:54.064307 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:21:54.065460 (+ 1153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:21:54.065463 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:21:54.065482 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:21:54.065565 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:54.065570 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:54.066941 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:54.066945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:54.067958 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:54.067967 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:54.069004 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:54.069006 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:54.069150 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:54.069194 (+ 44us) client_negotiation.cc:769] Sending connection context
0618 22:21:54.069304 (+ 110us) client_negotiation.cc:240] Negotiation successful
0618 22:21:54.069453 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"mutex_wait_us":23,"thread_start_us":92,"threads_started":1}
I20260618 22:21:54.069630 20083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:54.063325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:34657 (local address 127.0.0.1:55176)
0618 22:21:54.063689 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:54.063739 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0618 22:21:54.063861 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:21:54.064689 (+ 828us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:21:54.064696 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:21:54.064724 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:21:54.064876 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:54.064884 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:54.066491 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:54.066494 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:54.068109 (+ 1615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:54.068120 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:54.069092 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:54.069095 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:54.069185 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:54.069201 (+ 16us) client_negotiation.cc:769] Sending connection context
0618 22:21:54.069304 (+ 103us) client_negotiation.cc:240] Negotiation successful
0618 22:21:54.069453 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":198,"thread_start_us":109,"threads_started":1}
I20260618 22:21:54.070077 20087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:54.064140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59204 (local address 127.0.98.67:46305)
0618 22:21:54.065004 (+ 864us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:54.065016 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:54.065036 (+ 20us) server_negotiation.cc:407] Connection header received
0618 22:21:54.065108 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:54.065112 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:54.065297 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:54.065706 (+ 409us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:21:54.065842 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:54.066784 (+ 942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:54.068081 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:54.068893 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:54.069007 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:54.069164 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:54.069806 (+ 642us) server_negotiation.cc:299] Negotiation successful
0618 22:21:54.069930 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":751,"thread_start_us":109,"threads_started":1}
I20260618 22:21:54.070815 20086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:54.064067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:43787 (local address 127.0.0.1:37068)
0618 22:21:54.064676 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:54.064688 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 22:21:54.064762 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:21:54.065306 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:21:54.065309 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:21:54.065326 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:21:54.065415 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:54.065423 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:54.066423 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:54.066435 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:54.068109 (+ 1674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:54.068120 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:54.070418 (+ 2298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:54.070434 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:54.070521 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:54.070541 (+ 20us) client_negotiation.cc:769] Sending connection context
0618 22:21:54.070615 (+ 74us) client_negotiation.cc:240] Negotiation successful
0618 22:21:54.070689 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"mutex_wait_us":170,"spinlock_wait_cycles":1408,"thread_start_us":78,"threads_started":1}
I20260618 22:21:54.071208 20088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:54.064443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37068 (local address 127.0.98.66:43787)
0618 22:21:54.064942 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:54.064952 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:54.064974 (+ 22us) server_negotiation.cc:407] Connection header received
0618 22:21:54.065032 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:54.065036 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:54.065165 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:54.065316 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:21:54.065519 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:54.066285 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:54.069238 (+ 2953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:54.070092 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:54.070203 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:54.070352 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:54.071004 (+ 652us) server_negotiation.cc:299] Negotiation successful
0618 22:21:54.071098 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":117,"threads_started":1}
I20260618 22:21:54.073098 19738 tablet_service.cc:1467] Processing CreateTablet for tablet eea3b96666a8449b982a2eeedc6ab1d3 (DEFAULT_TABLE table=test-table [id=baa405af5512422a8ef01cfbd340fbda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:21:54.073217 20006 tablet_service.cc:1467] Processing CreateTablet for tablet eea3b96666a8449b982a2eeedc6ab1d3 (DEFAULT_TABLE table=test-table [id=baa405af5512422a8ef01cfbd340fbda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:21:54.073261 19872 tablet_service.cc:1467] Processing CreateTablet for tablet eea3b96666a8449b982a2eeedc6ab1d3 (DEFAULT_TABLE table=test-table [id=baa405af5512422a8ef01cfbd340fbda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:21:54.074190 19872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eea3b96666a8449b982a2eeedc6ab1d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:21:54.074199 19738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eea3b96666a8449b982a2eeedc6ab1d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:21:54.074194 20006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eea3b96666a8449b982a2eeedc6ab1d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:21:54.080034 20089 tablet_bootstrap.cc:492] T eea3b96666a8449b982a2eeedc6ab1d3 P 310921fdc75642cdb086aef7a2c75980: Bootstrap starting.
I20260618 22:21:54.081274 20091 tablet_bootstrap.cc:492] T eea3b96666a8449b982a2eeedc6ab1d3 P 4f4388a49b274f47be007b57c076597e: Bootstrap starting.
I20260618 22:21:54.081571 20090 tablet_bootstrap.cc:492] T eea3b96666a8449b982a2eeedc6ab1d3 P 717954a6fb284a85b2599f4705fafe42: Bootstrap starting.
I20260618 22:21:54.082952 20089 tablet_bootstrap.cc:654] T eea3b96666a8449b982a2eeedc6ab1d3 P 310921fdc75642cdb086aef7a2c75980: Neither blocks nor log segments found. Creating new log.
I20260618 22:21:54.083343 20091 tablet_bootstrap.cc:654] T eea3b96666a8449b982a2eeedc6ab1d3 P 4f4388a49b274f47be007b57c076597e: Neither blocks nor log segments found. Creating new log.
I20260618 22:21:54.083598 20090 tablet_bootstrap.cc:654] T eea3b96666a8449b982a2eeedc6ab1d3 P 717954a6fb284a85b2599f4705fafe42: Neither blocks nor log segments found. Creating new log.
I20260618 22:21:54.084087 20089 log.cc:824] T eea3b96666a8449b982a2eeedc6ab1d3 P 310921fdc75642cdb086aef7a2c75980: Log is configured to *not* fsync() on all Append() calls
I20260618 22:21:54.084461 20091 log.cc:824] T eea3b96666a8449b982a2eeedc6ab1d3 P 4f4388a49b274f47be007b57c076597e: Log is configured to *not* fsync() on all Append() calls
I20260618 22:21:54.084467 20090 log.cc:824] T eea3b96666a8449b982a2eeedc6ab1d3 P 717954a6fb284a85b2599f4705fafe42: Log is configured to *not* fsync() on all Append() calls
I20260618 22:21:54.086051 20089 tablet_bootstrap.cc:492] T eea3b96666a8449b982a2eeedc6ab1d3 P 310921fdc75642cdb086aef7a2c75980: No bootstrap required, opened a new log
I20260618 22:21:54.086241 20089 ts_tablet_manager.cc:1397] T eea3b96666a8449b982a2eeedc6ab1d3 P 310921fdc75642cdb086aef7a2c75980: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260618 22:21:54.086400 20090 tablet_bootstrap.cc:492] T eea3b96666a8449b982a2eeedc6ab1d3 P 717954a6fb284a85b2599f4705fafe42: No bootstrap required, opened a new log
I20260618 22:21:54.086575 20090 ts_tablet_manager.cc:1397] T eea3b96666a8449b982a2eeedc6ab1d3 P 717954a6fb284a85b2599f4705fafe42: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260618 22:21:54.086709 20091 tablet_bootstrap.cc:492] T eea3b96666a8449b982a2eeedc6ab1d3 P 4f4388a49b274f47be007b57c076597e: No bootstrap required, opened a new log
I20260618 22:21:54.086869 20091 ts_tablet_manager.cc:1397] T eea3b96666a8449b982a2eeedc6ab1d3 P 4f4388a49b274f47be007b57c076597e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260618 22:21:54.089459 20089 raft_consensus.cc:348] T eea3b96666a8449b982a2eeedc6ab1d3 P 310921fdc75642cdb086aef7a2c75980 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717954a6fb284a85b2599f4705fafe42" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34657 } } peers { permanent_uuid: "310921fdc75642cdb086aef7a2c75980" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43787 } } peers { permanent_uuid: "4f4388a49b274f47be007b57c076597e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46305 } }
I20260618 22:21:54.089710 20089 raft_consensus.cc:374] T eea3b96666a8449b982a2eeedc6ab1d3 P 310921fdc75642cdb086aef7a2c75980 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:21:54.089771 20089 raft_consensus.cc:729] T eea3b96666a8449b982a2eeedc6ab1d3 P 310921fdc75642cdb086aef7a2c75980 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 310921fdc75642cdb086aef7a2c75980, State: Initialized, Role: FOLLOWER
I20260618 22:21:54.090245 20089 consensus_queue.cc:260] T eea3b96666a8449b982a2eeedc6ab1d3 P 310921fdc75642cdb086aef7a2c75980 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717954a6fb284a85b2599f4705fafe42" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34657 } } peers { permanent_uuid: "310921fdc75642cdb086aef7a2c75980" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43787 } } peers { permanent_uuid: "4f4388a49b274f47be007b57c076597e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46305 } }
I20260618 22:21:54.090832 19938 heartbeater.cc:503] Master 127.0.98.126:34719 was elected leader, sending a full tablet report...
I20260618 22:21:54.090739 20090 raft_consensus.cc:348] T eea3b96666a8449b982a2eeedc6ab1d3 P 717954a6fb284a85b2599f4705fafe42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717954a6fb284a85b2599f4705fafe42" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34657 } } peers { permanent_uuid: "310921fdc75642cdb086aef7a2c75980" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43787 } } peers { permanent_uuid: "4f4388a49b274f47be007b57c076597e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46305 } }
I20260618 22:21:54.090975 20090 raft_consensus.cc:374] T eea3b96666a8449b982a2eeedc6ab1d3 P 717954a6fb284a85b2599f4705fafe42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:21:54.091028 20090 raft_consensus.cc:729] T eea3b96666a8449b982a2eeedc6ab1d3 P 717954a6fb284a85b2599f4705fafe42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 717954a6fb284a85b2599f4705fafe42, State: Initialized, Role: FOLLOWER
I20260618 22:21:54.091217 20089 ts_tablet_manager.cc:1428] T eea3b96666a8449b982a2eeedc6ab1d3 P 310921fdc75642cdb086aef7a2c75980: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260618 22:21:54.091405 20090 consensus_queue.cc:260] T eea3b96666a8449b982a2eeedc6ab1d3 P 717954a6fb284a85b2599f4705fafe42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717954a6fb284a85b2599f4705fafe42" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34657 } } peers { permanent_uuid: "310921fdc75642cdb086aef7a2c75980" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43787 } } peers { permanent_uuid: "4f4388a49b274f47be007b57c076597e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46305 } }
I20260618 22:21:54.090610 20091 raft_consensus.cc:348] T eea3b96666a8449b982a2eeedc6ab1d3 P 4f4388a49b274f47be007b57c076597e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717954a6fb284a85b2599f4705fafe42" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34657 } } peers { permanent_uuid: "310921fdc75642cdb086aef7a2c75980" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43787 } } peers { permanent_uuid: "4f4388a49b274f47be007b57c076597e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46305 } }
I20260618 22:21:54.091583 20091 raft_consensus.cc:374] T eea3b96666a8449b982a2eeedc6ab1d3 P 4f4388a49b274f47be007b57c076597e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:21:54.091645 20091 raft_consensus.cc:729] T eea3b96666a8449b982a2eeedc6ab1d3 P 4f4388a49b274f47be007b57c076597e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f4388a49b274f47be007b57c076597e, State: Initialized, Role: FOLLOWER
I20260618 22:21:54.092034 19804 heartbeater.cc:503] Master 127.0.98.126:34719 was elected leader, sending a full tablet report...
I20260618 22:21:54.092139 20091 consensus_queue.cc:260] T eea3b96666a8449b982a2eeedc6ab1d3 P 4f4388a49b274f47be007b57c076597e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717954a6fb284a85b2599f4705fafe42" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34657 } } peers { permanent_uuid: "310921fdc75642cdb086aef7a2c75980" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43787 } } peers { permanent_uuid: "4f4388a49b274f47be007b57c076597e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46305 } }
I20260618 22:21:54.092696 20072 heartbeater.cc:503] Master 127.0.98.126:34719 was elected leader, sending a full tablet report...
I20260618 22:21:54.093019 20091 ts_tablet_manager.cc:1428] T eea3b96666a8449b982a2eeedc6ab1d3 P 4f4388a49b274f47be007b57c076597e: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260618 22:21:54.093794 20090 ts_tablet_manager.cc:1428] T eea3b96666a8449b982a2eeedc6ab1d3 P 717954a6fb284a85b2599f4705fafe42: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260618 22:21:54.244956 20095 raft_consensus.cc:482] T eea3b96666a8449b982a2eeedc6ab1d3 P 310921fdc75642cdb086aef7a2c75980 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:21:54.245149 20095 raft_consensus.cc:504] T eea3b96666a8449b982a2eeedc6ab1d3 P 310921fdc75642cdb086aef7a2c75980 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717954a6fb284a85b2599f4705fafe42" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34657 } } peers { permanent_uuid: "310921fdc75642cdb086aef7a2c75980" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43787 } } peers { permanent_uuid: "4f4388a49b274f47be007b57c076597e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46305 } }
I20260618 22:21:54.246470 20095 leader_election.cc:288] T eea3b96666a8449b982a2eeedc6ab1d3 P 310921fdc75642cdb086aef7a2c75980 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 717954a6fb284a85b2599f4705fafe42 (127.0.98.65:34657), 4f4388a49b274f47be007b57c076597e (127.0.98.67:46305)
I20260618 22:21:54.250273 20084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:54.246826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:40021 (local address 127.0.98.65:34657)
0618 22:21:54.246993 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:54.246999 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:54.247246 (+ 247us) server_negotiation.cc:407] Connection header received
0618 22:21:54.247402 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:54.247406 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:54.247457 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:54.247539 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:21:54.247788 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:54.248283 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:54.249294 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:54.249687 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:54.249800 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:54.249871 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:54.250091 (+ 220us) server_negotiation.cc:299] Negotiation successful
0618 22:21:54.250154 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260618 22:21:54.250274 20087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:54.246864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:56673 (local address 127.0.98.67:46305)
0618 22:21:54.246997 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:54.247001 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:54.247107 (+ 106us) server_negotiation.cc:407] Connection header received
0618 22:21:54.247262 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:54.247265 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:54.247319 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:54.247403 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:21:54.247664 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:54.248163 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:54.249261 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:54.249669 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:54.249785 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:54.249845 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:54.250081 (+ 236us) server_negotiation.cc:299] Negotiation successful
0618 22:21:54.250156 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260618 22:21:54.250274 19940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:54.246737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:46305 (local address 127.0.98.66:56673)
0618 22:21:54.247005 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:54.247023 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0618 22:21:54.247158 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:21:54.247413 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:21:54.247415 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:21:54.247430 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:21:54.247559 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:54.247565 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:54.248274 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:54.248278 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:54.249145 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:54.249154 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:54.249799 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:54.249803 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:54.249948 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:54.249974 (+ 26us) client_negotiation.cc:769] Sending connection context
0618 22:21:54.250059 (+ 85us) client_negotiation.cc:240] Negotiation successful
0618 22:21:54.250126 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":6}
I20260618 22:21:54.250286 20098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:54.246746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:34657 (local address 127.0.98.66:40021)
0618 22:21:54.247153 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:54.247164 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 22:21:54.247285 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:21:54.247554 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:21:54.247557 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:21:54.247571 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:21:54.247686 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:54.247691 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:54.248398 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:54.248401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:54.249177 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:54.249185 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:54.249803 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:54.249806 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:54.249970 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:54.249989 (+ 19us) client_negotiation.cc:769] Sending connection context
0618 22:21:54.250076 (+ 87us) client_negotiation.cc:240] Negotiation successful
0618 22:21:54.250143 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"mutex_wait_us":40,"thread_start_us":122,"threads_started":1}
I20260618 22:21:54.250998 19758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eea3b96666a8449b982a2eeedc6ab1d3" candidate_uuid: "310921fdc75642cdb086aef7a2c75980" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "717954a6fb284a85b2599f4705fafe42" is_pre_election: true
I20260618 22:21:54.250993 20026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eea3b96666a8449b982a2eeedc6ab1d3" candidate_uuid: "310921fdc75642cdb086aef7a2c75980" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f4388a49b274f47be007b57c076597e" is_pre_election: true
I20260618 22:21:54.251317 20026 raft_consensus.cc:2454] T eea3b96666a8449b982a2eeedc6ab1d3 P 4f4388a49b274f47be007b57c076597e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 310921fdc75642cdb086aef7a2c75980 in term 0.
I20260618 22:21:54.251322 19758 raft_consensus.cc:2454] T eea3b96666a8449b982a2eeedc6ab1d3 P 717954a6fb284a85b2599f4705fafe42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 310921fdc75642cdb086aef7a2c75980 in term 0.
I20260618 22:21:54.251798 19826 leader_election.cc:302] T eea3b96666a8449b982a2eeedc6ab1d3 P 310921fdc75642cdb086aef7a2c75980 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 310921fdc75642cdb086aef7a2c75980, 717954a6fb284a85b2599f4705fafe42; no voters:
I20260618 22:21:54.252085 20095 raft_consensus.cc:2781] T eea3b96666a8449b982a2eeedc6ab1d3 P 310921fdc75642cdb086aef7a2c75980 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:21:54.252167 20095 raft_consensus.cc:482] T eea3b96666a8449b982a2eeedc6ab1d3 P 310921fdc75642cdb086aef7a2c75980 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:21:54.252207 20095 raft_consensus.cc:3037] T eea3b96666a8449b982a2eeedc6ab1d3 P 310921fdc75642cdb086aef7a2c75980 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:21:54.253176 20095 raft_consensus.cc:504] T eea3b96666a8449b982a2eeedc6ab1d3 P 310921fdc75642cdb086aef7a2c75980 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717954a6fb284a85b2599f4705fafe42" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34657 } } peers { permanent_uuid: "310921fdc75642cdb086aef7a2c75980" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43787 } } peers { permanent_uuid: "4f4388a49b274f47be007b57c076597e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46305 } }
I20260618 22:21:54.253495 20095 leader_election.cc:288] T eea3b96666a8449b982a2eeedc6ab1d3 P 310921fdc75642cdb086aef7a2c75980 [CANDIDATE]: Term 1 election: Requested vote from peers 717954a6fb284a85b2599f4705fafe42 (127.0.98.65:34657), 4f4388a49b274f47be007b57c076597e (127.0.98.67:46305)
I20260618 22:21:54.253803 19758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eea3b96666a8449b982a2eeedc6ab1d3" candidate_uuid: "310921fdc75642cdb086aef7a2c75980" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "717954a6fb284a85b2599f4705fafe42"
I20260618 22:21:54.253841 20026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eea3b96666a8449b982a2eeedc6ab1d3" candidate_uuid: "310921fdc75642cdb086aef7a2c75980" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f4388a49b274f47be007b57c076597e"
I20260618 22:21:54.253969 19758 raft_consensus.cc:3037] T eea3b96666a8449b982a2eeedc6ab1d3 P 717954a6fb284a85b2599f4705fafe42 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:21:54.254060 20026 raft_consensus.cc:3037] T eea3b96666a8449b982a2eeedc6ab1d3 P 4f4388a49b274f47be007b57c076597e [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:21:54.255085 19758 raft_consensus.cc:2454] T eea3b96666a8449b982a2eeedc6ab1d3 P 717954a6fb284a85b2599f4705fafe42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 310921fdc75642cdb086aef7a2c75980 in term 1.
I20260618 22:21:54.255225 20026 raft_consensus.cc:2454] T eea3b96666a8449b982a2eeedc6ab1d3 P 4f4388a49b274f47be007b57c076597e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 310921fdc75642cdb086aef7a2c75980 in term 1.
I20260618 22:21:54.255403 19826 leader_election.cc:302] T eea3b96666a8449b982a2eeedc6ab1d3 P 310921fdc75642cdb086aef7a2c75980 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 310921fdc75642cdb086aef7a2c75980, 717954a6fb284a85b2599f4705fafe42; no voters:
I20260618 22:21:54.255565 20095 raft_consensus.cc:2781] T eea3b96666a8449b982a2eeedc6ab1d3 P 310921fdc75642cdb086aef7a2c75980 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:21:54.255744 20095 raft_consensus.cc:686] T eea3b96666a8449b982a2eeedc6ab1d3 P 310921fdc75642cdb086aef7a2c75980 [term 1 LEADER]: Becoming Leader. State: Replica: 310921fdc75642cdb086aef7a2c75980, State: Running, Role: LEADER
I20260618 22:21:54.256068 20095 consensus_queue.cc:237] T eea3b96666a8449b982a2eeedc6ab1d3 P 310921fdc75642cdb086aef7a2c75980 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717954a6fb284a85b2599f4705fafe42" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34657 } } peers { permanent_uuid: "310921fdc75642cdb086aef7a2c75980" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43787 } } peers { permanent_uuid: "4f4388a49b274f47be007b57c076597e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46305 } }
I20260618 22:21:54.259114 19612 catalog_manager.cc:5495] T eea3b96666a8449b982a2eeedc6ab1d3 P 310921fdc75642cdb086aef7a2c75980 reported cstate change: term changed from 0 to 1, leader changed from <none> to 310921fdc75642cdb086aef7a2c75980 (127.0.98.66). New cstate: current_term: 1 leader_uuid: "310921fdc75642cdb086aef7a2c75980" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "717954a6fb284a85b2599f4705fafe42" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34657 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "310921fdc75642cdb086aef7a2c75980" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 43787 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4f4388a49b274f47be007b57c076597e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46305 } health_report { overall_health: UNKNOWN } } }
W20260618 22:21:54.263712 20073 tablet.cc:2369] T eea3b96666a8449b982a2eeedc6ab1d3 P 4f4388a49b274f47be007b57c076597e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260618 22:21:54.295044 19939 tablet.cc:2369] T eea3b96666a8449b982a2eeedc6ab1d3 P 310921fdc75642cdb086aef7a2c75980: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:21:54.306542 20088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:54.302630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37074 (local address 127.0.98.66:43787)
0618 22:21:54.302785 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:54.302790 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:54.302826 (+ 36us) server_negotiation.cc:407] Connection header received
0618 22:21:54.303039 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:54.303041 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:54.303103 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:54.303212 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:21:54.303504 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:54.304035 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:54.304938 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:54.305165 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:54.305259 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:54.305582 (+ 323us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:21:54.306081 (+ 499us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:21:54.306181 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:54.306315 (+ 134us) server_negotiation.cc:299] Negotiation successful
0618 22:21:54.306370 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260618 22:21:54.312143 20026 raft_consensus.cc:1261] T eea3b96666a8449b982a2eeedc6ab1d3 P 4f4388a49b274f47be007b57c076597e [term 1 FOLLOWER]: Refusing update from remote peer 310921fdc75642cdb086aef7a2c75980: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:21:54.312168 19758 raft_consensus.cc:1261] T eea3b96666a8449b982a2eeedc6ab1d3 P 717954a6fb284a85b2599f4705fafe42 [term 1 FOLLOWER]: Refusing update from remote peer 310921fdc75642cdb086aef7a2c75980: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:21:54.312788 20095 consensus_queue.cc:1035] T eea3b96666a8449b982a2eeedc6ab1d3 P 310921fdc75642cdb086aef7a2c75980 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f4388a49b274f47be007b57c076597e" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 46305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:21:54.312976 20099 consensus_queue.cc:1035] T eea3b96666a8449b982a2eeedc6ab1d3 P 310921fdc75642cdb086aef7a2c75980 [LEADER]: Connected to new peer: Peer: permanent_uuid: "717954a6fb284a85b2599f4705fafe42" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 34657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260618 22:21:54.317739 19805 tablet.cc:2369] T eea3b96666a8449b982a2eeedc6ab1d3 P 717954a6fb284a85b2599f4705fafe42: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:21:54.328310 19611 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:21:54.328498 19611 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:21:54.331014 19611 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d37a62d43de64edcb13dc3155ac51628: Sending DeleteTablet for 3 replicas of tablet eea3b96666a8449b982a2eeedc6ab1d3
I20260618 22:21:54.331887 19738 tablet_service.cc:1514] Processing DeleteTablet for tablet eea3b96666a8449b982a2eeedc6ab1d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:21:54 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:55176
I20260618 22:21:54.331815 19872 tablet_service.cc:1514] Processing DeleteTablet for tablet eea3b96666a8449b982a2eeedc6ab1d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:21:54 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:37068
I20260618 22:21:54.332350 20006 tablet_service.cc:1514] Processing DeleteTablet for tablet eea3b96666a8449b982a2eeedc6ab1d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:21:54 UTC) from {username='kudu', principal='kudu/127.0.98.126@KRBTEST.COM'} at 127.0.0.1:59204
I20260618 22:21:54.332549 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 19675
I20260618 22:21:54.332931 20113 tablet_replica.cc:330] T eea3b96666a8449b982a2eeedc6ab1d3 P 310921fdc75642cdb086aef7a2c75980: stopping tablet replica
I20260618 22:21:54.333416 20113 raft_consensus.cc:2229] T eea3b96666a8449b982a2eeedc6ab1d3 P 310921fdc75642cdb086aef7a2c75980 [term 1 LEADER]: Raft consensus shutting down.
I20260618 22:21:54.333838 20113 raft_consensus.cc:2258] T eea3b96666a8449b982a2eeedc6ab1d3 P 310921fdc75642cdb086aef7a2c75980 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:21:54.335218 20113 ts_tablet_manager.cc:1905] T eea3b96666a8449b982a2eeedc6ab1d3 P 310921fdc75642cdb086aef7a2c75980: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:21:54.338791 20114 tablet_replica.cc:330] T eea3b96666a8449b982a2eeedc6ab1d3 P 4f4388a49b274f47be007b57c076597e: stopping tablet replica
I20260618 22:21:54.339056 20113 ts_tablet_manager.cc:1918] T eea3b96666a8449b982a2eeedc6ab1d3 P 310921fdc75642cdb086aef7a2c75980: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:21:54.339166 20113 log.cc:1196] T eea3b96666a8449b982a2eeedc6ab1d3 P 310921fdc75642cdb086aef7a2c75980: Deleting WAL directory at /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/minicluster-data/ts-1/wal/wals/eea3b96666a8449b982a2eeedc6ab1d3
I20260618 22:21:54.339113 20114 raft_consensus.cc:2229] T eea3b96666a8449b982a2eeedc6ab1d3 P 4f4388a49b274f47be007b57c076597e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:21:54.339512 20113 ts_tablet_manager.cc:1939] T eea3b96666a8449b982a2eeedc6ab1d3 P 310921fdc75642cdb086aef7a2c75980: Deleting consensus metadata
I20260618 22:21:54.339618 20114 raft_consensus.cc:2258] T eea3b96666a8449b982a2eeedc6ab1d3 P 4f4388a49b274f47be007b57c076597e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:21:54.340677 19598 catalog_manager.cc:4841] TS 310921fdc75642cdb086aef7a2c75980 (127.0.98.66:43787): tablet eea3b96666a8449b982a2eeedc6ab1d3 (table test-table [id=baa405af5512422a8ef01cfbd340fbda]) successfully deleted
I20260618 22:21:54.341427 20114 ts_tablet_manager.cc:1905] T eea3b96666a8449b982a2eeedc6ab1d3 P 4f4388a49b274f47be007b57c076597e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260618 22:21:54.341964 19598 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.65:34657 (error 108)
I20260618 22:21:54.342137 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 19809
I20260618 22:21:54.344424 20086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:54.344155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:34657 (local address 127.0.0.1:55192)
0618 22:21:54.344282 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:54.344355 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.65:34657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260618 22:21:54.344620 19598 catalog_manager.cc:4584] TS 717954a6fb284a85b2599f4705fafe42 (127.0.98.65:34657): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet eea3b96666a8449b982a2eeedc6ab1d3: Network error: Client connection negotiation failed: client connection to 127.0.98.65:34657: connect: Connection refused (error 111)
I20260618 22:21:54.344681 20114 ts_tablet_manager.cc:1918] T eea3b96666a8449b982a2eeedc6ab1d3 P 4f4388a49b274f47be007b57c076597e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:21:54.344774 20114 log.cc:1196] T eea3b96666a8449b982a2eeedc6ab1d3 P 4f4388a49b274f47be007b57c076597e: Deleting WAL directory at /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781821065380026-393-0/minicluster-data/ts-2/wal/wals/eea3b96666a8449b982a2eeedc6ab1d3
I20260618 22:21:54.345104 20114 ts_tablet_manager.cc:1939] T eea3b96666a8449b982a2eeedc6ab1d3 P 4f4388a49b274f47be007b57c076597e: Deleting consensus metadata
I20260618 22:21:54.346239 19599 catalog_manager.cc:4841] TS 4f4388a49b274f47be007b57c076597e (127.0.98.67:46305): tablet eea3b96666a8449b982a2eeedc6ab1d3 (table test-table [id=baa405af5512422a8ef01cfbd340fbda]) successfully deleted
I20260618 22:21:54.348568 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 19943
I20260618 22:21:54.353698 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 19581
2026-06-18T22:21:54Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2411 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-18T22:21:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:21:54Z Disabled control of system clock
I20260618 22:21:54.383729 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821065380026-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821065380026-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821065380026-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:42533
--webserver_interface=127.0.98.126
--webserver_port=0
--builtin_ntp_servers=127.0.98.84:37927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:42533
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260618 22:21:54.493520 20119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:21:54.493744 20119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:21:54.493778 20119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:21:54.497704 20119 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:21:54.497753 20119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:21:54.497769 20119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:21:54.497786 20119 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:21:54.497802 20119 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:21:54.502411 20119 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:37927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821065380026-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821065380026-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:42533
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:42533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20119
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821065380026-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:21:54.503346 20119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:21:54.504632 20119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:21:54.513264 20124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:21:54.513249 20127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:21:54.513314 20119 server_base.cc:1034] running on GCE node
W20260618 22:21:54.513291 20125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:21:54.514019 20119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:21:54.515033 20119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:21:54.516201 20119 hybrid_clock.cc:648] HybridClock initialized: now 1781821314516172 us; error 41 us; skew 500 ppm
I20260618 22:21:54.518147 20119 webserver.cc:460] Webserver started at http://127.0.98.126:38557/ using document root <none> and password file <none>
I20260618 22:21:54.518680 20119 fs_manager.cc:359] Metadata directory not provided
I20260618 22:21:54.518740 20119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:21:54.518968 20119 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:21:54.520870 20119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821065380026-393-0/minicluster-data/master-0/data/instance:
uuid: "5dcbc1d176d243ae9778f6e0f77587af"
format_stamp: "Formatted at 2026-06-18 22:21:54 on dist-test-slave-g6r4"
server_key: "293c643338f4c17dd9fd82e98d73b579"
server_key_iv: "0606d4ce7c7b8e1bc112ab19eadf97dd"
server_key_version: "encryptionkey@0"
I20260618 22:21:54.521405 20119 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821065380026-393-0/minicluster-data/master-0/wal/instance:
uuid: "5dcbc1d176d243ae9778f6e0f77587af"
format_stamp: "Formatted at 2026-06-18 22:21:54 on dist-test-slave-g6r4"
server_key: "293c643338f4c17dd9fd82e98d73b579"
server_key_iv: "0606d4ce7c7b8e1bc112ab19eadf97dd"
server_key_version: "encryptionkey@0"
I20260618 22:21:54.524860 20119 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260618 22:21:54.527215 20133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:54.528398 20119 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260618 22:21:54.528528 20119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821065380026-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821065380026-393-0/minicluster-data/master-0/wal
uuid: "5dcbc1d176d243ae9778f6e0f77587af"
format_stamp: "Formatted at 2026-06-18 22:21:54 on dist-test-slave-g6r4"
server_key: "293c643338f4c17dd9fd82e98d73b579"
server_key_iv: "0606d4ce7c7b8e1bc112ab19eadf97dd"
server_key_version: "encryptionkey@0"
I20260618 22:21:54.528631 20119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821065380026-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821065380026-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821065380026-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:21:54.539557 20119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:21:54.540112 20119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:21:54.540293 20119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:21:54.547052 20119 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:42533
I20260618 22:21:54.547081 20185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:42533 every 8 connection(s)
I20260618 22:21:54.547917 20119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
I20260618 22:21:54.549196 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 20119
I20260618 22:21:54.549322 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821065380026-393-0/minicluster-data/master-0/wal/instance
I20260618 22:21:54.549664 393 external_mini_cluster.cc:1428] Setting key 03164e1912deeb57f3d7a8c3a7599f53
I20260618 22:21:54.551404 20186 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:21:54.556553 20186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5dcbc1d176d243ae9778f6e0f77587af: Bootstrap starting.
I20260618 22:21:54.558336 20189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:54.550971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38928 (local address 127.0.98.126:42533)
0618 22:21:54.551598 (+ 627us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:54.551613 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:54.551647 (+ 34us) server_negotiation.cc:407] Connection header received
0618 22:21:54.552252 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:54.552266 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:54.552621 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:54.552932 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:21:54.553351 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:54.554356 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:54.555212 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:54.555659 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:54.555833 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:54.556712 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:21:54.556717 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:21:54.556732 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:21:54.557098 (+ 366us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:21:54.557151 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:21:54.557171 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:21:54.557275 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:54.557387 (+ 112us) server_negotiation.cc:299] Negotiation successful
0618 22:21:54.557623 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":126,"threads_started":1}
I20260618 22:21:54.558772 20186 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5dcbc1d176d243ae9778f6e0f77587af: Neither blocks nor log segments found. Creating new log.
I20260618 22:21:54.559623 20186 log.cc:824] T 00000000000000000000000000000000 P 5dcbc1d176d243ae9778f6e0f77587af: Log is configured to *not* fsync() on all Append() calls
I20260618 22:21:54.561415 20186 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5dcbc1d176d243ae9778f6e0f77587af: No bootstrap required, opened a new log
I20260618 22:21:54.564616 20186 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5dcbc1d176d243ae9778f6e0f77587af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dcbc1d176d243ae9778f6e0f77587af" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42533 } }
I20260618 22:21:54.564858 20186 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5dcbc1d176d243ae9778f6e0f77587af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:21:54.564896 20186 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5dcbc1d176d243ae9778f6e0f77587af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dcbc1d176d243ae9778f6e0f77587af, State: Initialized, Role: FOLLOWER
I20260618 22:21:54.565272 20186 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5dcbc1d176d243ae9778f6e0f77587af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dcbc1d176d243ae9778f6e0f77587af" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42533 } }
I20260618 22:21:54.565382 20186 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5dcbc1d176d243ae9778f6e0f77587af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:21:54.565420 20186 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5dcbc1d176d243ae9778f6e0f77587af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:21:54.565487 20186 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5dcbc1d176d243ae9778f6e0f77587af [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:21:54.566521 20186 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5dcbc1d176d243ae9778f6e0f77587af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dcbc1d176d243ae9778f6e0f77587af" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42533 } }
I20260618 22:21:54.566816 20186 leader_election.cc:302] T 00000000000000000000000000000000 P 5dcbc1d176d243ae9778f6e0f77587af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5dcbc1d176d243ae9778f6e0f77587af; no voters:
I20260618 22:21:54.567090 20186 leader_election.cc:288] T 00000000000000000000000000000000 P 5dcbc1d176d243ae9778f6e0f77587af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:21:54.567185 20191 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5dcbc1d176d243ae9778f6e0f77587af [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:21:54.567413 20191 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5dcbc1d176d243ae9778f6e0f77587af [term 1 LEADER]: Becoming Leader. State: Replica: 5dcbc1d176d243ae9778f6e0f77587af, State: Running, Role: LEADER
I20260618 22:21:54.567723 20191 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5dcbc1d176d243ae9778f6e0f77587af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dcbc1d176d243ae9778f6e0f77587af" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42533 } }
I20260618 22:21:54.568204 20186 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5dcbc1d176d243ae9778f6e0f77587af [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:21:54.569339 20192 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dcbc1d176d243ae9778f6e0f77587af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5dcbc1d176d243ae9778f6e0f77587af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dcbc1d176d243ae9778f6e0f77587af" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42533 } } }
I20260618 22:21:54.569505 20192 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dcbc1d176d243ae9778f6e0f77587af [sys.catalog]: This master's current role is: LEADER
I20260618 22:21:54.569340 20193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dcbc1d176d243ae9778f6e0f77587af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5dcbc1d176d243ae9778f6e0f77587af. Latest consensus state: current_term: 1 leader_uuid: "5dcbc1d176d243ae9778f6e0f77587af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5dcbc1d176d243ae9778f6e0f77587af" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 42533 } } }
I20260618 22:21:54.569739 20193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dcbc1d176d243ae9778f6e0f77587af [sys.catalog]: This master's current role is: LEADER
I20260618 22:21:54.570501 20200 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:21:54.573015 20200 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:21:54.578347 20200 catalog_manager.cc:1347] Generated new cluster ID: fe9efff6e6094e9c8106aaf148affa5e
I20260618 22:21:54.578411 20200 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:21:54.587819 20200 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:21:54.588645 20200 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:21:54.598660 20200 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5dcbc1d176d243ae9778f6e0f77587af: Generated new TSK 0
I20260618 22:21:54.599470 20200 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:21:54.609633 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821065380026-393-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821065380026-393-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821065380026-393-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.65:0
--local_ip_for_outbound_sockets=127.0.98.65
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:42533
--builtin_ntp_servers=127.0.98.84:37927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260618 22:21:54.718961 20210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:21:54.719343 20210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:21:54.719403 20210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:21:54.723855 20210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:21:54.723905 20210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:21:54.723965 20210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.65
I20260618 22:21:54.729069 20210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:37927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821065380026-393-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821065380026-393-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.65
--webserver_port=0
--tserver_master_addrs=127.0.98.126:42533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.65
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821065380026-393-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:21:54.730166 20210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:21:54.731518 20210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:21:54.740896 20215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:21:54.741000 20210 server_base.cc:1034] running on GCE node
W20260618 22:21:54.740931 20216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:21:54.740896 20218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:21:54.741533 20210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:21:54.742159 20210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:21:54.743319 20210 hybrid_clock.cc:648] HybridClock initialized: now 1781821314743301 us; error 30 us; skew 500 ppm
I20260618 22:21:54.745102 20210 webserver.cc:460] Webserver started at http://127.0.98.65:37217/ using document root <none> and password file <none>
I20260618 22:21:54.745632 20210 fs_manager.cc:359] Metadata directory not provided
I20260618 22:21:54.745689 20210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:21:54.745990 20210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:21:54.747994 20210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821065380026-393-0/minicluster-data/ts-0/data/instance:
uuid: "25d0776c5b7341b893f9ce6c7090bf11"
format_stamp: "Formatted at 2026-06-18 22:21:54 on dist-test-slave-g6r4"
server_key: "0c11506f943b43d679232f9f1c153549"
server_key_iv: "cf8aa0adb12b4c4b3dd95ea07dd77dd8"
server_key_version: "encryptionkey@0"
I20260618 22:21:54.748530 20210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821065380026-393-0/minicluster-data/ts-0/wal/instance:
uuid: "25d0776c5b7341b893f9ce6c7090bf11"
format_stamp: "Formatted at 2026-06-18 22:21:54 on dist-test-slave-g6r4"
server_key: "0c11506f943b43d679232f9f1c153549"
server_key_iv: "cf8aa0adb12b4c4b3dd95ea07dd77dd8"
server_key_version: "encryptionkey@0"
I20260618 22:21:54.751916 20210 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260618 22:21:54.754222 20224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:54.755439 20210 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260618 22:21:54.755560 20210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821065380026-393-0/minicluster-data/ts-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821065380026-393-0/minicluster-data/ts-0/wal
uuid: "25d0776c5b7341b893f9ce6c7090bf11"
format_stamp: "Formatted at 2026-06-18 22:21:54 on dist-test-slave-g6r4"
server_key: "0c11506f943b43d679232f9f1c153549"
server_key_iv: "cf8aa0adb12b4c4b3dd95ea07dd77dd8"
server_key_version: "encryptionkey@0"
I20260618 22:21:54.755663 20210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821065380026-393-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821065380026-393-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821065380026-393-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:21:54.771679 20210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:21:54.772318 20210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:21:54.772514 20210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:21:54.773077 20210 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:21:54.774041 20210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:21:54.774091 20210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:54.774132 20210 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:21:54.774168 20210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:54.782547 20210 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.65:38645
I20260618 22:21:54.782567 20337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.65:38645 every 8 connection(s)
I20260618 22:21:54.783427 20210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821065380026-393-0/minicluster-data/ts-0/data/info.pb
I20260618 22:21:54.785629 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 20210
I20260618 22:21:54.785763 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821065380026-393-0/minicluster-data/ts-0/wal/instance
I20260618 22:21:54.786137 393 external_mini_cluster.cc:1428] Setting key 263b7a45be1169fc530905b5363f1f63
I20260618 22:21:54.788079 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821065380026-393-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821065380026-393-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821065380026-393-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.66:0
--local_ip_for_outbound_sockets=127.0.98.66
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:42533
--builtin_ntp_servers=127.0.98.84:37927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260618 22:21:54.793320 20189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:54.785175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:40181 (local address 127.0.98.126:42533)
0618 22:21:54.785308 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:54.785312 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:54.786662 (+ 1350us) server_negotiation.cc:407] Connection header received
0618 22:21:54.788381 (+ 1719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:54.788387 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:54.788454 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:54.788552 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:21:54.789417 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:54.790045 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:54.791184 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:54.791510 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:54.791637 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:54.792738 (+ 1101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:21:54.792741 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:21:54.792744 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:21:54.792757 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:21:54.792804 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:21:54.792813 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:21:54.792891 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:54.793090 (+ 199us) server_negotiation.cc:299] Negotiation successful
0618 22:21:54.793167 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260618 22:21:54.794194 20340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:54.785686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:42533 (local address 127.0.98.65:40181)
0618 22:21:54.786472 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:54.786542 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0618 22:21:54.788082 (+ 1540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:21:54.788700 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:21:54.788707 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:21:54.788790 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:21:54.789277 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:54.789289 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:54.790168 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:54.790171 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:54.791055 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:54.791065 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:54.791672 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:54.791676 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:54.791841 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:54.792482 (+ 641us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 22:21:54.792484 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:21:54.792522 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 22:21:54.792585 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:21:54.792905 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:21:54.792915 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:21:54.792928 (+ 13us) client_negotiation.cc:769] Sending connection context
0618 22:21:54.793089 (+ 161us) client_negotiation.cc:240] Negotiation successful
0618 22:21:54.793363 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":187,"threads_started":1}
I20260618 22:21:54.795405 20338 heartbeater.cc:347] Connected to a master server at 127.0.98.126:42533
I20260618 22:21:54.795668 20338 heartbeater.cc:464] Registering TS with master...
I20260618 22:21:54.796320 20338 heartbeater.cc:511] Master 127.0.98.126:42533 requested a full tablet report, sending...
I20260618 22:21:54.798067 20150 ts_manager.cc:194] Registered new tserver with Master: 25d0776c5b7341b893f9ce6c7090bf11 (127.0.98.65:38645)
I20260618 22:21:54.799651 20150 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.65:40181
W20260618 22:21:54.896399 20341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:21:54.896643 20341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:21:54.896680 20341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:21:54.903074 20341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:21:54.903151 20341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:21:54.903236 20341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.66
I20260618 22:21:54.908409 20341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:37927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821065380026-393-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821065380026-393-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.66
--webserver_port=0
--tserver_master_addrs=127.0.98.126:42533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.66
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821065380026-393-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:21:54.909488 20341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:21:54.910897 20341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:21:54.920367 20346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:21:54.920432 20347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:21:54.920372 20349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:21:54.920590 20341 server_base.cc:1034] running on GCE node
I20260618 22:21:54.921013 20341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:21:54.921674 20341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:21:54.922852 20341 hybrid_clock.cc:648] HybridClock initialized: now 1781821314922829 us; error 33 us; skew 500 ppm
I20260618 22:21:54.924728 20341 webserver.cc:460] Webserver started at http://127.0.98.66:34539/ using document root <none> and password file <none>
I20260618 22:21:54.925312 20341 fs_manager.cc:359] Metadata directory not provided
I20260618 22:21:54.925375 20341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:21:54.925623 20341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:21:54.927758 20341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821065380026-393-0/minicluster-data/ts-1/data/instance:
uuid: "7ac2b54d80784132b44ba43e5a30a3d1"
format_stamp: "Formatted at 2026-06-18 22:21:54 on dist-test-slave-g6r4"
server_key: "80fd5064b23d344854ea4e53f6ecd51f"
server_key_iv: "928cee7345a74170b1d47d67244e5a1b"
server_key_version: "encryptionkey@0"
I20260618 22:21:54.928355 20341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821065380026-393-0/minicluster-data/ts-1/wal/instance:
uuid: "7ac2b54d80784132b44ba43e5a30a3d1"
format_stamp: "Formatted at 2026-06-18 22:21:54 on dist-test-slave-g6r4"
server_key: "80fd5064b23d344854ea4e53f6ecd51f"
server_key_iv: "928cee7345a74170b1d47d67244e5a1b"
server_key_version: "encryptionkey@0"
I20260618 22:21:54.932220 20341 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260618 22:21:54.934685 20355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:54.935895 20341 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:21:54.936025 20341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821065380026-393-0/minicluster-data/ts-1/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821065380026-393-0/minicluster-data/ts-1/wal
uuid: "7ac2b54d80784132b44ba43e5a30a3d1"
format_stamp: "Formatted at 2026-06-18 22:21:54 on dist-test-slave-g6r4"
server_key: "80fd5064b23d344854ea4e53f6ecd51f"
server_key_iv: "928cee7345a74170b1d47d67244e5a1b"
server_key_version: "encryptionkey@0"
I20260618 22:21:54.936136 20341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821065380026-393-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821065380026-393-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821065380026-393-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:21:54.985881 20341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:21:54.986709 20341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:21:54.986936 20341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:21:54.987594 20341 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:21:54.988616 20341 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:21:54.988667 20341 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:54.988765 20341 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:21:54.988809 20341 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:54.998343 20341 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.66:37163
I20260618 22:21:54.998370 20468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.66:37163 every 8 connection(s)
I20260618 22:21:54.999270 20341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821065380026-393-0/minicluster-data/ts-1/data/info.pb
I20260618 22:21:55.004312 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 20341
I20260618 22:21:55.004446 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821065380026-393-0/minicluster-data/ts-1/wal/instance
I20260618 22:21:55.004752 393 external_mini_cluster.cc:1428] Setting key aad77a4e98171e627ec06479dcc6ff35
I20260618 22:21:55.006803 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821065380026-393-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821065380026-393-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821065380026-393-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.67:0
--local_ip_for_outbound_sockets=127.0.98.67
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:42533
--builtin_ntp_servers=127.0.98.84:37927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260618 22:21:55.008980 20189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:55.001047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.66:59793 (local address 127.0.98.126:42533)
0618 22:21:55.001188 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:55.001192 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:55.002292 (+ 1100us) server_negotiation.cc:407] Connection header received
0618 22:21:55.003620 (+ 1328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:55.003622 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:55.003674 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:55.003746 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:21:55.004944 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:55.005604 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:55.006955 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:55.007211 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:55.007324 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:55.008443 (+ 1119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:21:55.008449 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:21:55.008453 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:21:55.008469 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:21:55.008496 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:21:55.008509 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:21:55.008603 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:55.008742 (+ 139us) server_negotiation.cc:299] Negotiation successful
0618 22:21:55.008814 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260618 22:21:55.009817 20471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:55.001431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:42533 (local address 127.0.98.66:59793)
0618 22:21:55.002138 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:55.002179 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0618 22:21:55.003397 (+ 1218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:21:55.003914 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:21:55.003920 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:21:55.004004 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:21:55.004727 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:55.004747 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:55.005732 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:55.005735 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:55.006825 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:55.006834 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:55.007361 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:55.007365 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:55.007546 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:55.008184 (+ 638us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 22:21:55.008185 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:21:55.008226 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 22:21:55.008288 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:21:55.008608 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:21:55.008616 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:21:55.008620 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 22:21:55.008718 (+ 98us) client_negotiation.cc:240] Negotiation successful
0618 22:21:55.008939 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":117,"threads_started":1}
I20260618 22:21:55.011003 20469 heartbeater.cc:347] Connected to a master server at 127.0.98.126:42533
I20260618 22:21:55.011261 20469 heartbeater.cc:464] Registering TS with master...
I20260618 22:21:55.011879 20469 heartbeater.cc:511] Master 127.0.98.126:42533 requested a full tablet report, sending...
I20260618 22:21:55.012987 20150 ts_manager.cc:194] Registered new tserver with Master: 7ac2b54d80784132b44ba43e5a30a3d1 (127.0.98.66:37163)
I20260618 22:21:55.013588 20150 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.66:59793
W20260618 22:21:55.116184 20472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:21:55.116515 20472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:21:55.116580 20472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:21:55.120663 20472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:21:55.120715 20472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:21:55.120769 20472 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.67
I20260618 22:21:55.125370 20472 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:37927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821065380026-393-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821065380026-393-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.67
--webserver_port=0
--tserver_master_addrs=127.0.98.126:42533
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.67
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821065380026-393-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:21:55.126377 20472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:21:55.127628 20472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:21:55.136960 20477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:21:55.137003 20480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:21:55.137163 20472 server_base.cc:1034] running on GCE node
W20260618 22:21:55.136996 20478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:21:55.137609 20472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:21:55.138207 20472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:21:55.139395 20472 hybrid_clock.cc:648] HybridClock initialized: now 1781821315139388 us; error 31 us; skew 500 ppm
I20260618 22:21:55.141213 20472 webserver.cc:460] Webserver started at http://127.0.98.67:46757/ using document root <none> and password file <none>
I20260618 22:21:55.141765 20472 fs_manager.cc:359] Metadata directory not provided
I20260618 22:21:55.141855 20472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:21:55.142098 20472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:21:55.144057 20472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821065380026-393-0/minicluster-data/ts-2/data/instance:
uuid: "298c817fd03b430fa1b3d2b261dc7886"
format_stamp: "Formatted at 2026-06-18 22:21:55 on dist-test-slave-g6r4"
server_key: "f569546163dc059219b326ffdd194e46"
server_key_iv: "d2127ab33d0a0b63182dff24892059e3"
server_key_version: "encryptionkey@0"
I20260618 22:21:55.144594 20472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821065380026-393-0/minicluster-data/ts-2/wal/instance:
uuid: "298c817fd03b430fa1b3d2b261dc7886"
format_stamp: "Formatted at 2026-06-18 22:21:55 on dist-test-slave-g6r4"
server_key: "f569546163dc059219b326ffdd194e46"
server_key_iv: "d2127ab33d0a0b63182dff24892059e3"
server_key_version: "encryptionkey@0"
I20260618 22:21:55.148085 20472 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260618 22:21:55.150496 20486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:55.151660 20472 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:21:55.151798 20472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821065380026-393-0/minicluster-data/ts-2/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821065380026-393-0/minicluster-data/ts-2/wal
uuid: "298c817fd03b430fa1b3d2b261dc7886"
format_stamp: "Formatted at 2026-06-18 22:21:55 on dist-test-slave-g6r4"
server_key: "f569546163dc059219b326ffdd194e46"
server_key_iv: "d2127ab33d0a0b63182dff24892059e3"
server_key_version: "encryptionkey@0"
I20260618 22:21:55.151906 20472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821065380026-393-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821065380026-393-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821065380026-393-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:21:55.164201 20472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:21:55.164805 20472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:21:55.164989 20472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:21:55.165526 20472 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 22:21:55.166467 20472 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 22:21:55.166535 20472 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:55.166602 20472 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 22:21:55.166651 20472 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:55.175273 20472 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.67:37385
I20260618 22:21:55.175307 20599 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.67:37385 every 8 connection(s)
I20260618 22:21:55.176126 20472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821065380026-393-0/minicluster-data/ts-2/data/info.pb
I20260618 22:21:55.183064 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 20472
I20260618 22:21:55.183188 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821065380026-393-0/minicluster-data/ts-2/wal/instance
I20260618 22:21:55.183485 393 external_mini_cluster.cc:1428] Setting key df437e4b49f62fb833990cd5f733646c
I20260618 22:21:55.185350 20189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:55.177811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.67:37595 (local address 127.0.98.126:42533)
0618 22:21:55.177969 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:55.177972 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:55.179005 (+ 1033us) server_negotiation.cc:407] Connection header received
0618 22:21:55.180319 (+ 1314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:55.180321 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:55.180365 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:55.180445 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:21:55.181279 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:55.181889 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:55.183184 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:55.183463 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:55.183540 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:55.184887 (+ 1347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:21:55.184895 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:21:55.184899 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:21:55.184913 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:21:55.184936 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:21:55.184947 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:21:55.185024 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:55.185167 (+ 143us) server_negotiation.cc:299] Negotiation successful
0618 22:21:55.185219 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260618 22:21:55.186265 20602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:55.178252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:42533 (local address 127.0.98.67:37595)
0618 22:21:55.178853 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:55.178892 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0618 22:21:55.180135 (+ 1243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:21:55.180594 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:21:55.180601 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:21:55.180680 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:21:55.181160 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:55.181173 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:55.182058 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:55.182061 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:55.183045 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:55.183058 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:55.183618 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:55.183623 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:55.183818 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:55.184571 (+ 753us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 22:21:55.184573 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:21:55.184625 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 22:21:55.184706 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:21:55.185038 (+ 332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:21:55.185045 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:21:55.185049 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 22:21:55.185144 (+ 95us) client_negotiation.cc:240] Negotiation successful
0618 22:21:55.185343 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":108,"threads_started":1}
I20260618 22:21:55.187255 20600 heartbeater.cc:347] Connected to a master server at 127.0.98.126:42533
I20260618 22:21:55.187465 20600 heartbeater.cc:464] Registering TS with master...
I20260618 22:21:55.187924 20600 heartbeater.cc:511] Master 127.0.98.126:42533 requested a full tablet report, sending...
I20260618 22:21:55.188853 20150 ts_manager.cc:194] Registered new tserver with Master: 298c817fd03b430fa1b3d2b261dc7886 (127.0.98.67:37385)
I20260618 22:21:55.189378 20150 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.67:37595
I20260618 22:21:55.197201 393 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 22:21:55.202447 20189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:55.199115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38938 (local address 127.0.98.126:42533)
0618 22:21:55.199228 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:55.199232 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:55.199351 (+ 119us) server_negotiation.cc:407] Connection header received
0618 22:21:55.199440 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:55.199442 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:55.199476 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:55.199537 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:21:55.199789 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:55.200290 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:55.201073 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:55.201272 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:55.201360 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:55.202093 (+ 733us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:21:55.202095 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:21:55.202097 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:21:55.202107 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:21:55.202124 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:21:55.202131 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:21:55.202187 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:55.202257 (+ 70us) server_negotiation.cc:299] Negotiation successful
0618 22:21:55.202304 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260618 22:21:55.205163 20150 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38938:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 22:21:55.208173 20150 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 22:21:55.221680 20611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:55.216101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:37163 (local address 127.0.0.1:44836)
0618 22:21:55.216586 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:55.216633 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0618 22:21:55.216757 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:21:55.217749 (+ 992us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:21:55.217758 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:21:55.217790 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:21:55.218007 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:55.218015 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:55.219179 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:55.219183 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:55.220369 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:55.220377 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:55.221141 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:55.221145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:55.221255 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:55.221291 (+ 36us) client_negotiation.cc:769] Sending connection context
0618 22:21:55.221405 (+ 114us) client_negotiation.cc:240] Negotiation successful
0618 22:21:55.221522 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"mutex_wait_us":23,"spinlock_wait_cycles":1792,"thread_start_us":69,"threads_started":1}
I20260618 22:21:55.221683 20612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:55.216236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44836 (local address 127.0.98.66:37163)
0618 22:21:55.217229 (+ 993us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:55.217240 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:55.217266 (+ 26us) server_negotiation.cc:407] Connection header received
0618 22:21:55.217350 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:55.217356 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:55.217573 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:55.217730 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:21:55.218139 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:55.219065 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:55.220485 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:55.221018 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:55.221143 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:55.221332 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:55.221434 (+ 102us) server_negotiation.cc:299] Negotiation successful
0618 22:21:55.221538 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":877,"thread_start_us":147,"threads_started":1}
I20260618 22:21:55.222137 20613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:55.216585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:38645 (local address 127.0.0.1:53528)
0618 22:21:55.217137 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:55.217151 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:21:55.217239 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:21:55.217927 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:21:55.217930 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:21:55.217946 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:21:55.218105 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:55.218114 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:55.219696 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:55.219702 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:55.220671 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:55.220680 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:55.221711 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:55.221715 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:55.221835 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:55.221857 (+ 22us) client_negotiation.cc:769] Sending connection context
0618 22:21:55.221926 (+ 69us) client_negotiation.cc:240] Negotiation successful
0618 22:21:55.221987 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"mutex_wait_us":48,"thread_start_us":91,"threads_started":1}
I20260618 22:21:55.222889 20615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:55.216761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53528 (local address 127.0.98.65:38645)
0618 22:21:55.217476 (+ 715us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:55.217487 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:55.217506 (+ 19us) server_negotiation.cc:407] Connection header received
0618 22:21:55.217589 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:55.217594 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:55.217757 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:55.217930 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:21:55.218236 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:55.219432 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:55.220801 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:55.221581 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:55.222331 (+ 750us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:55.222545 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:55.222598 (+ 53us) server_negotiation.cc:299] Negotiation successful
0618 22:21:55.222718 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":117,"threads_started":1}
I20260618 22:21:55.223682 20614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:55.216770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:37385 (local address 127.0.0.1:42646)
0618 22:21:55.217693 (+ 923us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:55.217707 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:21:55.217799 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:21:55.218376 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:21:55.218379 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:21:55.218397 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:21:55.218481 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:55.218487 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:55.219768 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:55.219771 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:55.222049 (+ 2278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:55.222058 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:55.223261 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:55.223266 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:55.223364 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:55.223388 (+ 24us) client_negotiation.cc:769] Sending connection context
0618 22:21:55.223454 (+ 66us) client_negotiation.cc:240] Negotiation successful
0618 22:21:55.223523 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":847,"mutex_wait_us":249,"spinlock_wait_cycles":6784,"thread_start_us":92,"threads_started":1}
I20260618 22:21:55.224099 20616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:55.217126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42646 (local address 127.0.98.67:37385)
0618 22:21:55.217547 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:55.217556 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:55.217888 (+ 332us) server_negotiation.cc:407] Connection header received
0618 22:21:55.217989 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:55.217996 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:55.218228 (+ 232us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:55.218378 (+ 150us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:21:55.218604 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:55.219654 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:55.222167 (+ 2513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:55.222904 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:55.223012 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:55.223200 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:55.223838 (+ 638us) server_negotiation.cc:299] Negotiation successful
0618 22:21:55.223934 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":127,"threads_started":1}
I20260618 22:21:55.224947 20403 tablet_service.cc:1467] Processing CreateTablet for tablet 938f39357cad4e90a508b33cebd32c27 (DEFAULT_TABLE table=test-table [id=e03fd7f572ac4833a3a51cb772ed52b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:21:55.225816 20403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 938f39357cad4e90a508b33cebd32c27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:21:55.225430 20272 tablet_service.cc:1467] Processing CreateTablet for tablet 938f39357cad4e90a508b33cebd32c27 (DEFAULT_TABLE table=test-table [id=e03fd7f572ac4833a3a51cb772ed52b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:21:55.226449 20272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 938f39357cad4e90a508b33cebd32c27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:21:55.226732 20534 tablet_service.cc:1467] Processing CreateTablet for tablet 938f39357cad4e90a508b33cebd32c27 (DEFAULT_TABLE table=test-table [id=e03fd7f572ac4833a3a51cb772ed52b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 22:21:55.227669 20534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 938f39357cad4e90a508b33cebd32c27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:21:55.232103 20617 tablet_bootstrap.cc:492] T 938f39357cad4e90a508b33cebd32c27 P 7ac2b54d80784132b44ba43e5a30a3d1: Bootstrap starting.
I20260618 22:21:55.234338 20618 tablet_bootstrap.cc:492] T 938f39357cad4e90a508b33cebd32c27 P 25d0776c5b7341b893f9ce6c7090bf11: Bootstrap starting.
I20260618 22:21:55.234405 20617 tablet_bootstrap.cc:654] T 938f39357cad4e90a508b33cebd32c27 P 7ac2b54d80784132b44ba43e5a30a3d1: Neither blocks nor log segments found. Creating new log.
I20260618 22:21:55.235881 20617 log.cc:824] T 938f39357cad4e90a508b33cebd32c27 P 7ac2b54d80784132b44ba43e5a30a3d1: Log is configured to *not* fsync() on all Append() calls
I20260618 22:21:55.236503 20618 tablet_bootstrap.cc:654] T 938f39357cad4e90a508b33cebd32c27 P 25d0776c5b7341b893f9ce6c7090bf11: Neither blocks nor log segments found. Creating new log.
I20260618 22:21:55.236675 20619 tablet_bootstrap.cc:492] T 938f39357cad4e90a508b33cebd32c27 P 298c817fd03b430fa1b3d2b261dc7886: Bootstrap starting.
I20260618 22:21:55.237339 20618 log.cc:824] T 938f39357cad4e90a508b33cebd32c27 P 25d0776c5b7341b893f9ce6c7090bf11: Log is configured to *not* fsync() on all Append() calls
I20260618 22:21:55.237815 20617 tablet_bootstrap.cc:492] T 938f39357cad4e90a508b33cebd32c27 P 7ac2b54d80784132b44ba43e5a30a3d1: No bootstrap required, opened a new log
I20260618 22:21:55.238036 20617 ts_tablet_manager.cc:1397] T 938f39357cad4e90a508b33cebd32c27 P 7ac2b54d80784132b44ba43e5a30a3d1: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260618 22:21:55.238775 20619 tablet_bootstrap.cc:654] T 938f39357cad4e90a508b33cebd32c27 P 298c817fd03b430fa1b3d2b261dc7886: Neither blocks nor log segments found. Creating new log.
I20260618 22:21:55.239418 20618 tablet_bootstrap.cc:492] T 938f39357cad4e90a508b33cebd32c27 P 25d0776c5b7341b893f9ce6c7090bf11: No bootstrap required, opened a new log
I20260618 22:21:55.239604 20618 ts_tablet_manager.cc:1397] T 938f39357cad4e90a508b33cebd32c27 P 25d0776c5b7341b893f9ce6c7090bf11: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260618 22:21:55.239646 20619 log.cc:824] T 938f39357cad4e90a508b33cebd32c27 P 298c817fd03b430fa1b3d2b261dc7886: Log is configured to *not* fsync() on all Append() calls
I20260618 22:21:55.241577 20619 tablet_bootstrap.cc:492] T 938f39357cad4e90a508b33cebd32c27 P 298c817fd03b430fa1b3d2b261dc7886: No bootstrap required, opened a new log
I20260618 22:21:55.241750 20619 ts_tablet_manager.cc:1397] T 938f39357cad4e90a508b33cebd32c27 P 298c817fd03b430fa1b3d2b261dc7886: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260618 22:21:55.242290 20617 raft_consensus.cc:348] T 938f39357cad4e90a508b33cebd32c27 P 7ac2b54d80784132b44ba43e5a30a3d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac2b54d80784132b44ba43e5a30a3d1" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37163 } } peers { permanent_uuid: "298c817fd03b430fa1b3d2b261dc7886" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 37385 } } peers { permanent_uuid: "25d0776c5b7341b893f9ce6c7090bf11" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 38645 } }
I20260618 22:21:55.242527 20617 raft_consensus.cc:374] T 938f39357cad4e90a508b33cebd32c27 P 7ac2b54d80784132b44ba43e5a30a3d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:21:55.242578 20617 raft_consensus.cc:729] T 938f39357cad4e90a508b33cebd32c27 P 7ac2b54d80784132b44ba43e5a30a3d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ac2b54d80784132b44ba43e5a30a3d1, State: Initialized, Role: FOLLOWER
I20260618 22:21:55.243010 20617 consensus_queue.cc:260] T 938f39357cad4e90a508b33cebd32c27 P 7ac2b54d80784132b44ba43e5a30a3d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac2b54d80784132b44ba43e5a30a3d1" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37163 } } peers { permanent_uuid: "298c817fd03b430fa1b3d2b261dc7886" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 37385 } } peers { permanent_uuid: "25d0776c5b7341b893f9ce6c7090bf11" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 38645 } }
I20260618 22:21:55.243856 20617 ts_tablet_manager.cc:1428] T 938f39357cad4e90a508b33cebd32c27 P 7ac2b54d80784132b44ba43e5a30a3d1: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260618 22:21:55.243768 20618 raft_consensus.cc:348] T 938f39357cad4e90a508b33cebd32c27 P 25d0776c5b7341b893f9ce6c7090bf11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac2b54d80784132b44ba43e5a30a3d1" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37163 } } peers { permanent_uuid: "298c817fd03b430fa1b3d2b261dc7886" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 37385 } } peers { permanent_uuid: "25d0776c5b7341b893f9ce6c7090bf11" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 38645 } }
I20260618 22:21:55.244015 20618 raft_consensus.cc:374] T 938f39357cad4e90a508b33cebd32c27 P 25d0776c5b7341b893f9ce6c7090bf11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:21:55.244086 20618 raft_consensus.cc:729] T 938f39357cad4e90a508b33cebd32c27 P 25d0776c5b7341b893f9ce6c7090bf11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 25d0776c5b7341b893f9ce6c7090bf11, State: Initialized, Role: FOLLOWER
I20260618 22:21:55.244231 20469 heartbeater.cc:503] Master 127.0.98.126:42533 was elected leader, sending a full tablet report...
I20260618 22:21:55.244485 20618 consensus_queue.cc:260] T 938f39357cad4e90a508b33cebd32c27 P 25d0776c5b7341b893f9ce6c7090bf11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac2b54d80784132b44ba43e5a30a3d1" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37163 } } peers { permanent_uuid: "298c817fd03b430fa1b3d2b261dc7886" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 37385 } } peers { permanent_uuid: "25d0776c5b7341b893f9ce6c7090bf11" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 38645 } }
I20260618 22:21:55.245234 20338 heartbeater.cc:503] Master 127.0.98.126:42533 was elected leader, sending a full tablet report...
I20260618 22:21:55.245409 20618 ts_tablet_manager.cc:1428] T 938f39357cad4e90a508b33cebd32c27 P 25d0776c5b7341b893f9ce6c7090bf11: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260618 22:21:55.245790 20619 raft_consensus.cc:348] T 938f39357cad4e90a508b33cebd32c27 P 298c817fd03b430fa1b3d2b261dc7886 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac2b54d80784132b44ba43e5a30a3d1" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37163 } } peers { permanent_uuid: "298c817fd03b430fa1b3d2b261dc7886" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 37385 } } peers { permanent_uuid: "25d0776c5b7341b893f9ce6c7090bf11" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 38645 } }
I20260618 22:21:55.246065 20619 raft_consensus.cc:374] T 938f39357cad4e90a508b33cebd32c27 P 298c817fd03b430fa1b3d2b261dc7886 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:21:55.246122 20619 raft_consensus.cc:729] T 938f39357cad4e90a508b33cebd32c27 P 298c817fd03b430fa1b3d2b261dc7886 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 298c817fd03b430fa1b3d2b261dc7886, State: Initialized, Role: FOLLOWER
I20260618 22:21:55.246526 20619 consensus_queue.cc:260] T 938f39357cad4e90a508b33cebd32c27 P 298c817fd03b430fa1b3d2b261dc7886 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac2b54d80784132b44ba43e5a30a3d1" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37163 } } peers { permanent_uuid: "298c817fd03b430fa1b3d2b261dc7886" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 37385 } } peers { permanent_uuid: "25d0776c5b7341b893f9ce6c7090bf11" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 38645 } }
I20260618 22:21:55.247169 20600 heartbeater.cc:503] Master 127.0.98.126:42533 was elected leader, sending a full tablet report...
I20260618 22:21:55.247493 20619 ts_tablet_manager.cc:1428] T 938f39357cad4e90a508b33cebd32c27 P 298c817fd03b430fa1b3d2b261dc7886: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
W20260618 22:21:55.250128 20470 tablet.cc:2369] T 938f39357cad4e90a508b33cebd32c27 P 7ac2b54d80784132b44ba43e5a30a3d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260618 22:21:55.284576 20339 tablet.cc:2369] T 938f39357cad4e90a508b33cebd32c27 P 25d0776c5b7341b893f9ce6c7090bf11: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:21:55.358079 20624 raft_consensus.cc:482] T 938f39357cad4e90a508b33cebd32c27 P 25d0776c5b7341b893f9ce6c7090bf11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 22:21:55.358282 20624 raft_consensus.cc:504] T 938f39357cad4e90a508b33cebd32c27 P 25d0776c5b7341b893f9ce6c7090bf11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac2b54d80784132b44ba43e5a30a3d1" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37163 } } peers { permanent_uuid: "298c817fd03b430fa1b3d2b261dc7886" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 37385 } } peers { permanent_uuid: "25d0776c5b7341b893f9ce6c7090bf11" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 38645 } }
I20260618 22:21:55.359367 20624 leader_election.cc:288] T 938f39357cad4e90a508b33cebd32c27 P 25d0776c5b7341b893f9ce6c7090bf11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7ac2b54d80784132b44ba43e5a30a3d1 (127.0.98.66:37163), 298c817fd03b430fa1b3d2b261dc7886 (127.0.98.67:37385)
I20260618 22:21:55.363636 20612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:55.359757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:55177 (local address 127.0.98.66:37163)
0618 22:21:55.359904 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:55.359907 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:55.360057 (+ 150us) server_negotiation.cc:407] Connection header received
0618 22:21:55.360211 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:55.360215 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:55.360270 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:55.360359 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:21:55.360814 (+ 455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:55.361427 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:55.362629 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:55.363043 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:55.363124 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:55.363172 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:55.363439 (+ 267us) server_negotiation.cc:299] Negotiation successful
0618 22:21:55.363512 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260618 22:21:55.363642 20626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:55.359670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.66:37163 (local address 127.0.98.65:55177)
0618 22:21:55.359945 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:55.359961 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 22:21:55.360088 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:21:55.360498 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:21:55.360501 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:21:55.360518 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:21:55.360668 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:55.360675 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:55.361535 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:55.361538 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:55.362499 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:55.362509 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:55.363169 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:55.363173 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:55.363273 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:55.363318 (+ 45us) client_negotiation.cc:769] Sending connection context
0618 22:21:55.363393 (+ 75us) client_negotiation.cc:240] Negotiation successful
0618 22:21:55.363476 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":3,"thread_start_us":118,"threads_started":1}
I20260618 22:21:55.363669 20616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:55.360284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.65:43119 (local address 127.0.98.67:37385)
0618 22:21:55.360448 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:55.360452 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:55.360603 (+ 151us) server_negotiation.cc:407] Connection header received
0618 22:21:55.360685 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:55.360688 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:55.360741 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:55.360814 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 22:21:55.361086 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:55.361606 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:55.362698 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:55.363125 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:55.363229 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:55.363287 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:55.363495 (+ 208us) server_negotiation.cc:299] Negotiation successful
0618 22:21:55.363563 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260618 22:21:55.363677 20627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:55.360140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.67:37385 (local address 127.0.98.65:43119)
0618 22:21:55.360511 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:55.360522 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 22:21:55.360609 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:21:55.360844 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:21:55.360847 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:21:55.360863 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 22:21:55.360980 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:55.360986 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:55.361718 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:55.361720 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:55.362540 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:21:55.362550 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:21:55.363272 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:21:55.363275 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:21:55.363348 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:55.363364 (+ 16us) client_negotiation.cc:769] Sending connection context
0618 22:21:55.363460 (+ 96us) client_negotiation.cc:240] Negotiation successful
0618 22:21:55.363525 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":89,"threads_started":1}
I20260618 22:21:55.364498 20554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "938f39357cad4e90a508b33cebd32c27" candidate_uuid: "25d0776c5b7341b893f9ce6c7090bf11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "298c817fd03b430fa1b3d2b261dc7886" is_pre_election: true
I20260618 22:21:55.364500 20423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "938f39357cad4e90a508b33cebd32c27" candidate_uuid: "25d0776c5b7341b893f9ce6c7090bf11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac2b54d80784132b44ba43e5a30a3d1" is_pre_election: true
I20260618 22:21:55.364784 20423 raft_consensus.cc:2454] T 938f39357cad4e90a508b33cebd32c27 P 7ac2b54d80784132b44ba43e5a30a3d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25d0776c5b7341b893f9ce6c7090bf11 in term 0.
I20260618 22:21:55.364791 20554 raft_consensus.cc:2454] T 938f39357cad4e90a508b33cebd32c27 P 298c817fd03b430fa1b3d2b261dc7886 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 25d0776c5b7341b893f9ce6c7090bf11 in term 0.
I20260618 22:21:55.365255 20228 leader_election.cc:302] T 938f39357cad4e90a508b33cebd32c27 P 25d0776c5b7341b893f9ce6c7090bf11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25d0776c5b7341b893f9ce6c7090bf11, 298c817fd03b430fa1b3d2b261dc7886; no voters:
I20260618 22:21:55.365538 20624 raft_consensus.cc:2781] T 938f39357cad4e90a508b33cebd32c27 P 25d0776c5b7341b893f9ce6c7090bf11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 22:21:55.365612 20624 raft_consensus.cc:482] T 938f39357cad4e90a508b33cebd32c27 P 25d0776c5b7341b893f9ce6c7090bf11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 22:21:55.365643 20624 raft_consensus.cc:3037] T 938f39357cad4e90a508b33cebd32c27 P 25d0776c5b7341b893f9ce6c7090bf11 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:21:55.366672 20624 raft_consensus.cc:504] T 938f39357cad4e90a508b33cebd32c27 P 25d0776c5b7341b893f9ce6c7090bf11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac2b54d80784132b44ba43e5a30a3d1" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37163 } } peers { permanent_uuid: "298c817fd03b430fa1b3d2b261dc7886" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 37385 } } peers { permanent_uuid: "25d0776c5b7341b893f9ce6c7090bf11" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 38645 } }
I20260618 22:21:55.366995 20624 leader_election.cc:288] T 938f39357cad4e90a508b33cebd32c27 P 25d0776c5b7341b893f9ce6c7090bf11 [CANDIDATE]: Term 1 election: Requested vote from peers 7ac2b54d80784132b44ba43e5a30a3d1 (127.0.98.66:37163), 298c817fd03b430fa1b3d2b261dc7886 (127.0.98.67:37385)
I20260618 22:21:55.367317 20423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "938f39357cad4e90a508b33cebd32c27" candidate_uuid: "25d0776c5b7341b893f9ce6c7090bf11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7ac2b54d80784132b44ba43e5a30a3d1"
I20260618 22:21:55.367399 20554 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "938f39357cad4e90a508b33cebd32c27" candidate_uuid: "25d0776c5b7341b893f9ce6c7090bf11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "298c817fd03b430fa1b3d2b261dc7886"
I20260618 22:21:55.367442 20423 raft_consensus.cc:3037] T 938f39357cad4e90a508b33cebd32c27 P 7ac2b54d80784132b44ba43e5a30a3d1 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:21:55.367542 20554 raft_consensus.cc:3037] T 938f39357cad4e90a508b33cebd32c27 P 298c817fd03b430fa1b3d2b261dc7886 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:21:55.368755 20423 raft_consensus.cc:2454] T 938f39357cad4e90a508b33cebd32c27 P 7ac2b54d80784132b44ba43e5a30a3d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25d0776c5b7341b893f9ce6c7090bf11 in term 1.
I20260618 22:21:55.368940 20554 raft_consensus.cc:2454] T 938f39357cad4e90a508b33cebd32c27 P 298c817fd03b430fa1b3d2b261dc7886 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 25d0776c5b7341b893f9ce6c7090bf11 in term 1.
I20260618 22:21:55.369129 20228 leader_election.cc:302] T 938f39357cad4e90a508b33cebd32c27 P 25d0776c5b7341b893f9ce6c7090bf11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 25d0776c5b7341b893f9ce6c7090bf11, 7ac2b54d80784132b44ba43e5a30a3d1; no voters:
I20260618 22:21:55.369306 20624 raft_consensus.cc:2781] T 938f39357cad4e90a508b33cebd32c27 P 25d0776c5b7341b893f9ce6c7090bf11 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:21:55.369519 20624 raft_consensus.cc:686] T 938f39357cad4e90a508b33cebd32c27 P 25d0776c5b7341b893f9ce6c7090bf11 [term 1 LEADER]: Becoming Leader. State: Replica: 25d0776c5b7341b893f9ce6c7090bf11, State: Running, Role: LEADER
I20260618 22:21:55.369849 20624 consensus_queue.cc:237] T 938f39357cad4e90a508b33cebd32c27 P 25d0776c5b7341b893f9ce6c7090bf11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac2b54d80784132b44ba43e5a30a3d1" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37163 } } peers { permanent_uuid: "298c817fd03b430fa1b3d2b261dc7886" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 37385 } } peers { permanent_uuid: "25d0776c5b7341b893f9ce6c7090bf11" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 38645 } }
I20260618 22:21:55.372946 20149 catalog_manager.cc:5495] T 938f39357cad4e90a508b33cebd32c27 P 25d0776c5b7341b893f9ce6c7090bf11 reported cstate change: term changed from 0 to 1, leader changed from <none> to 25d0776c5b7341b893f9ce6c7090bf11 (127.0.98.65). New cstate: current_term: 1 leader_uuid: "25d0776c5b7341b893f9ce6c7090bf11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7ac2b54d80784132b44ba43e5a30a3d1" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "298c817fd03b430fa1b3d2b261dc7886" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 37385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "25d0776c5b7341b893f9ce6c7090bf11" member_type: VOTER last_known_addr { host: "127.0.98.65" port: 38645 } health_report { overall_health: HEALTHY } } }
I20260618 22:21:55.413831 20615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:55.410423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53530 (local address 127.0.98.65:38645)
0618 22:21:55.410587 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:55.410591 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:55.410604 (+ 13us) server_negotiation.cc:407] Connection header received
0618 22:21:55.410678 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:55.410680 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:55.410732 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:55.410858 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:21:55.411049 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:55.411539 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:55.412353 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:55.412573 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:55.412652 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:55.413011 (+ 359us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:21:55.413434 (+ 423us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:21:55.413536 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:55.413592 (+ 56us) server_negotiation.cc:299] Negotiation successful
0618 22:21:55.413637 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260618 22:21:55.419561 20554 raft_consensus.cc:1261] T 938f39357cad4e90a508b33cebd32c27 P 298c817fd03b430fa1b3d2b261dc7886 [term 1 FOLLOWER]: Refusing update from remote peer 25d0776c5b7341b893f9ce6c7090bf11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:21:55.419561 20423 raft_consensus.cc:1261] T 938f39357cad4e90a508b33cebd32c27 P 7ac2b54d80784132b44ba43e5a30a3d1 [term 1 FOLLOWER]: Refusing update from remote peer 25d0776c5b7341b893f9ce6c7090bf11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:21:55.420325 20624 consensus_queue.cc:1035] T 938f39357cad4e90a508b33cebd32c27 P 25d0776c5b7341b893f9ce6c7090bf11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "298c817fd03b430fa1b3d2b261dc7886" member_type: VOTER last_known_addr { host: "127.0.98.67" port: 37385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:21:55.420573 20628 consensus_queue.cc:1035] T 938f39357cad4e90a508b33cebd32c27 P 25d0776c5b7341b893f9ce6c7090bf11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7ac2b54d80784132b44ba43e5a30a3d1" member_type: VOTER last_known_addr { host: "127.0.98.66" port: 37163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260618 22:21:55.427192 20601 tablet.cc:2369] T 938f39357cad4e90a508b33cebd32c27 P 298c817fd03b430fa1b3d2b261dc7886: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:21:55.436098 20148 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38938:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:21:55.436268 20148 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38938:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 22:21:55.438516 20148 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5dcbc1d176d243ae9778f6e0f77587af: Sending DeleteTablet for 3 replicas of tablet 938f39357cad4e90a508b33cebd32c27
I20260618 22:21:55.439199 20272 tablet_service.cc:1514] Processing DeleteTablet for tablet 938f39357cad4e90a508b33cebd32c27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:21:55 UTC) from {username='slave'} at 127.0.0.1:53528
I20260618 22:21:55.439448 20403 tablet_service.cc:1514] Processing DeleteTablet for tablet 938f39357cad4e90a508b33cebd32c27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:21:55 UTC) from {username='slave'} at 127.0.0.1:44836
I20260618 22:21:55.439666 20534 tablet_service.cc:1514] Processing DeleteTablet for tablet 938f39357cad4e90a508b33cebd32c27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 22:21:55 UTC) from {username='slave'} at 127.0.0.1:42646
I20260618 22:21:55.439934 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 20210
I20260618 22:21:55.440006 20642 tablet_replica.cc:330] T 938f39357cad4e90a508b33cebd32c27 P 7ac2b54d80784132b44ba43e5a30a3d1: stopping tablet replica
I20260618 22:21:55.440354 20642 raft_consensus.cc:2229] T 938f39357cad4e90a508b33cebd32c27 P 7ac2b54d80784132b44ba43e5a30a3d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:21:55.440672 20642 raft_consensus.cc:2258] T 938f39357cad4e90a508b33cebd32c27 P 7ac2b54d80784132b44ba43e5a30a3d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:21:55.440995 20643 tablet_replica.cc:330] T 938f39357cad4e90a508b33cebd32c27 P 298c817fd03b430fa1b3d2b261dc7886: stopping tablet replica
I20260618 22:21:55.441306 20643 raft_consensus.cc:2229] T 938f39357cad4e90a508b33cebd32c27 P 298c817fd03b430fa1b3d2b261dc7886 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 22:21:55.441553 20643 raft_consensus.cc:2258] T 938f39357cad4e90a508b33cebd32c27 P 298c817fd03b430fa1b3d2b261dc7886 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 22:21:55.442046 20642 ts_tablet_manager.cc:1905] T 938f39357cad4e90a508b33cebd32c27 P 7ac2b54d80784132b44ba43e5a30a3d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:21:55.442842 20643 ts_tablet_manager.cc:1905] T 938f39357cad4e90a508b33cebd32c27 P 298c817fd03b430fa1b3d2b261dc7886: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 22:21:55.444746 20642 ts_tablet_manager.cc:1918] T 938f39357cad4e90a508b33cebd32c27 P 7ac2b54d80784132b44ba43e5a30a3d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:21:55.444839 20642 log.cc:1196] T 938f39357cad4e90a508b33cebd32c27 P 7ac2b54d80784132b44ba43e5a30a3d1: Deleting WAL directory at /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821065380026-393-0/minicluster-data/ts-1/wal/wals/938f39357cad4e90a508b33cebd32c27
I20260618 22:21:55.445143 20642 ts_tablet_manager.cc:1939] T 938f39357cad4e90a508b33cebd32c27 P 7ac2b54d80784132b44ba43e5a30a3d1: Deleting consensus metadata
I20260618 22:21:55.445587 20643 ts_tablet_manager.cc:1918] T 938f39357cad4e90a508b33cebd32c27 P 298c817fd03b430fa1b3d2b261dc7886: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 22:21:55.445678 20643 log.cc:1196] T 938f39357cad4e90a508b33cebd32c27 P 298c817fd03b430fa1b3d2b261dc7886: Deleting WAL directory at /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781821065380026-393-0/minicluster-data/ts-2/wal/wals/938f39357cad4e90a508b33cebd32c27
I20260618 22:21:55.446163 20643 ts_tablet_manager.cc:1939] T 938f39357cad4e90a508b33cebd32c27 P 298c817fd03b430fa1b3d2b261dc7886: Deleting consensus metadata
I20260618 22:21:55.446455 20137 catalog_manager.cc:4841] TS 7ac2b54d80784132b44ba43e5a30a3d1 (127.0.98.66:37163): tablet 938f39357cad4e90a508b33cebd32c27 (table test-table [id=e03fd7f572ac4833a3a51cb772ed52b1]) successfully deleted
W20260618 22:21:55.447357 20134 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.65:38645 (error 108)
I20260618 22:21:55.447409 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 20341
I20260618 22:21:55.447372 20137 catalog_manager.cc:4841] TS 298c817fd03b430fa1b3d2b261dc7886 (127.0.98.67:37385): tablet 938f39357cad4e90a508b33cebd32c27 (table test-table [id=e03fd7f572ac4833a3a51cb772ed52b1]) successfully deleted
I20260618 22:21:55.450659 20614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:55.450397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.65:38645 (local address 127.0.0.1:53542)
0618 22:21:55.450528 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0618 22:21:55.450590 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.65:38645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260618 22:21:55.450871 20134 catalog_manager.cc:4584] TS 25d0776c5b7341b893f9ce6c7090bf11 (127.0.98.65:38645): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 938f39357cad4e90a508b33cebd32c27: Network error: Client connection negotiation failed: client connection to 127.0.98.65:38645: connect: Connection refused (error 111)
I20260618 22:21:55.452880 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 20472
I20260618 22:21:55.458333 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 20119
2026-06-18T22:21:55Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1103 ms)
[----------] 6 tests from EncryptionPolicyTest (10751 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781821065380026-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:21:55 dist-test-slave-g6r4 krb5kdc[20647](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:21:55 dist-test-slave-g6r4 krb5kdc[20647](info): set up 2 sockets
Jun 18 22:21:55 dist-test-slave-g6r4 krb5kdc[20647](info): commencing operation
krb5kdc: starting...
W20260618 22:21:56.497991 393 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781821065380026-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781821065380026-393-0/krb5kdc/test-user.keytab.
Jun 18 22:21:56 dist-test-slave-g6r4 krb5kdc[20647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:21:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:21:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781821065380026-393-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781821065380026-393-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260618 22:21:56.616400 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781821065380026-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781821065380026-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781821065380026-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45263
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781821065380026-393-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41789
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45263
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:21:56.727537 20660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:21:56.727877 20660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:21:56.727942 20660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:21:56.732568 20660 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:21:56.732631 20660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:21:56.732661 20660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:21:56.732677 20660 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:21:56.732693 20660 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:21:56.738113 20660 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41789
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781821065380026-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781821065380026-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45263
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781821065380026-393-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20660
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781821065380026-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:21:56.739074 20660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:21:56.740037 20660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:21:56.748435 20665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:21:56.748430 20668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:21:56.748606 20660 server_base.cc:1034] running on GCE node
W20260618 22:21:56.748425 20666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:21:56.749289 20660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:21:56.750397 20660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:21:56.751559 20660 hybrid_clock.cc:648] HybridClock initialized: now 1781821316751545 us; error 24 us; skew 500 ppm
Jun 18 22:21:56 dist-test-slave-g6r4 krb5kdc[20647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821316, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:21:56.754181 20660 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260618 22:21:56.754937 20660 webserver.cc:460] Webserver started at http://127.0.0.1:37013/ using document root <none> and password file <none>
I20260618 22:21:56.755425 20660 fs_manager.cc:359] Metadata directory not provided
I20260618 22:21:56.755481 20660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:21:56.755635 20660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:21:56.757500 20660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781821065380026-393-0/minicluster-data/master-0/data/instance:
uuid: "65b4b9ccfb3b44d698179269b858515e"
format_stamp: "Formatted at 2026-06-18 22:21:56 on dist-test-slave-g6r4"
server_key: "5051de0b6308d54a217c83c05aad917b"
server_key_iv: "2f6b39685a97edd144f97c004eacf4d8"
server_key_version: "encryptionkey@0"
I20260618 22:21:56.758069 20660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781821065380026-393-0/minicluster-data/master-0/wal/instance:
uuid: "65b4b9ccfb3b44d698179269b858515e"
format_stamp: "Formatted at 2026-06-18 22:21:56 on dist-test-slave-g6r4"
server_key: "5051de0b6308d54a217c83c05aad917b"
server_key_iv: "2f6b39685a97edd144f97c004eacf4d8"
server_key_version: "encryptionkey@0"
I20260618 22:21:56.761526 20660 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:21:56.763888 20675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:56.765053 20660 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:21:56.765197 20660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781821065380026-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781821065380026-393-0/minicluster-data/master-0/wal
uuid: "65b4b9ccfb3b44d698179269b858515e"
format_stamp: "Formatted at 2026-06-18 22:21:56 on dist-test-slave-g6r4"
server_key: "5051de0b6308d54a217c83c05aad917b"
server_key_iv: "2f6b39685a97edd144f97c004eacf4d8"
server_key_version: "encryptionkey@0"
I20260618 22:21:56.765307 20660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781821065380026-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781821065380026-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781821065380026-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:21:56.781065 20660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:21:56.784116 20660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:21:56.784324 20660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:21:56.791285 20660 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45263
I20260618 22:21:56.791316 20727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45263 every 8 connection(s)
I20260618 22:21:56.792210 20660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
I20260618 22:21:56.792450 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 20660
I20260618 22:21:56.792598 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781821065380026-393-0/minicluster-data/master-0/wal/instance
I20260618 22:21:56.792928 393 external_mini_cluster.cc:1428] Setting key 7a7bf4214922ff600b56a9ea7087bb51
I20260618 22:21:56.795295 20728 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:21:56.800654 20728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65b4b9ccfb3b44d698179269b858515e: Bootstrap starting.
Jun 18 22:21:56 dist-test-slave-g6r4 krb5kdc[20647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260618 22:21:56.803045 20728 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65b4b9ccfb3b44d698179269b858515e: Neither blocks nor log segments found. Creating new log.
I20260618 22:21:56.803740 20728 log.cc:824] T 00000000000000000000000000000000 P 65b4b9ccfb3b44d698179269b858515e: Log is configured to *not* fsync() on all Append() calls
I20260618 22:21:56.805467 20728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65b4b9ccfb3b44d698179269b858515e: No bootstrap required, opened a new log
I20260618 22:21:56.806527 20731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:56.794307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54937 (local address 127.0.0.1:45263)
0618 22:21:56.794713 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:56.794727 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:56.794759 (+ 32us) server_negotiation.cc:407] Connection header received
0618 22:21:56.795364 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:56.795379 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:56.795750 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:56.796069 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:21:56.796872 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:56.797847 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:56.798727 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:56.799186 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:56.799360 (+ 174us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:56.801822 (+ 2462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:21:56.801827 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:21:56.801838 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:21:56.801862 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:21:56.803709 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:21:56.804253 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:21:56.804257 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:21:56.804262 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:21:56.804369 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:21:56.804710 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:21:56.804715 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:21:56.804721 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:21:56.805078 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:21:56.805290 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:56.805585 (+ 295us) server_negotiation.cc:299] Negotiation successful
0618 22:21:56.805788 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":140,"threads_started":1}
I20260618 22:21:56.809398 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:21:56.809573 20728 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65b4b9ccfb3b44d698179269b858515e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65b4b9ccfb3b44d698179269b858515e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45263 } }
I20260618 22:21:56.809796 20728 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65b4b9ccfb3b44d698179269b858515e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:21:56.809891 20728 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65b4b9ccfb3b44d698179269b858515e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65b4b9ccfb3b44d698179269b858515e, State: Initialized, Role: FOLLOWER
I20260618 22:21:56.810338 20728 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65b4b9ccfb3b44d698179269b858515e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65b4b9ccfb3b44d698179269b858515e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45263 } }
I20260618 22:21:56.810480 20728 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65b4b9ccfb3b44d698179269b858515e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:21:56.810547 20728 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65b4b9ccfb3b44d698179269b858515e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:21:56.810639 20728 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65b4b9ccfb3b44d698179269b858515e [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:21:56.811561 20728 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65b4b9ccfb3b44d698179269b858515e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65b4b9ccfb3b44d698179269b858515e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45263 } }
I20260618 22:21:56.811895 20728 leader_election.cc:302] T 00000000000000000000000000000000 P 65b4b9ccfb3b44d698179269b858515e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65b4b9ccfb3b44d698179269b858515e; no voters:
I20260618 22:21:56.812188 20728 leader_election.cc:288] T 00000000000000000000000000000000 P 65b4b9ccfb3b44d698179269b858515e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:21:56.812309 20738 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65b4b9ccfb3b44d698179269b858515e [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:21:56.812541 20738 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65b4b9ccfb3b44d698179269b858515e [term 1 LEADER]: Becoming Leader. State: Replica: 65b4b9ccfb3b44d698179269b858515e, State: Running, Role: LEADER
I20260618 22:21:56.813292 20728 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65b4b9ccfb3b44d698179269b858515e [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:21:56.813692 20738 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65b4b9ccfb3b44d698179269b858515e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65b4b9ccfb3b44d698179269b858515e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45263 } }
I20260618 22:21:56.815395 20739 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65b4b9ccfb3b44d698179269b858515e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65b4b9ccfb3b44d698179269b858515e. Latest consensus state: current_term: 1 leader_uuid: "65b4b9ccfb3b44d698179269b858515e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65b4b9ccfb3b44d698179269b858515e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45263 } } }
I20260618 22:21:56.816025 20739 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65b4b9ccfb3b44d698179269b858515e [sys.catalog]: This master's current role is: LEADER
I20260618 22:21:56.815585 20740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65b4b9ccfb3b44d698179269b858515e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65b4b9ccfb3b44d698179269b858515e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65b4b9ccfb3b44d698179269b858515e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45263 } } }
I20260618 22:21:56.816267 20740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65b4b9ccfb3b44d698179269b858515e [sys.catalog]: This master's current role is: LEADER
W20260618 22:21:56.818284 20752 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 65b4b9ccfb3b44d698179269b858515e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 22:21:56.818400 20752 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 22:21:56.818512 20753 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:21:56.819434 20753 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:21:56.820618 20731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:56.811479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33939 (local address 127.0.0.1:45263)
0618 22:21:56.811606 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:56.811610 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:56.811752 (+ 142us) server_negotiation.cc:407] Connection header received
0618 22:21:56.811838 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:56.811843 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:56.811890 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:56.811973 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:21:56.812787 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:56.813406 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:56.814495 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:56.814871 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:56.814994 (+ 123us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:56.816897 (+ 1903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:21:56.816900 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:21:56.816903 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:21:56.816926 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:21:56.818672 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:21:56.819235 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:21:56.819239 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:21:56.819240 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:21:56.819323 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:21:56.819725 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:21:56.819728 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:21:56.819730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:21:56.819929 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:21:56.820077 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:56.820390 (+ 313us) server_negotiation.cc:299] Negotiation successful
0618 22:21:56.820480 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260618 22:21:56.825387 20753 catalog_manager.cc:1347] Generated new cluster ID: 0ea0edafa17b43779494c8c20e5d70af
I20260618 22:21:56.825446 20753 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:21:56.833555 20753 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:21:56.834751 20753 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:21:56.839748 20753 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65b4b9ccfb3b44d698179269b858515e: Generated new TSK 0
I20260618 22:21:56.840438 20753 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:21:56.845394 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 20660
W20260618 22:21:56.850541 20645 connection.cc:535] client connection to 127.0.0.1:45263 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45263): Connection reset by peer (error 104)
2026-06-18T22:21:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1385 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781821065380026-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:21:56 dist-test-slave-g6r4 krb5kdc[20759](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:21:56 dist-test-slave-g6r4 krb5kdc[20759](info): set up 2 sockets
Jun 18 22:21:56 dist-test-slave-g6r4 krb5kdc[20759](info): commencing operation
krb5kdc: starting...
W20260618 22:21:57.885988 393 mini_kdc.cc:121] Time spent starting KDC: real 1.021s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781821065380026-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781821065380026-393-0/krb5kdc/test-user.keytab.
Jun 18 22:21:57 dist-test-slave-g6r4 krb5kdc[20759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:21:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:21:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781821065380026-393-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781821065380026-393-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260618 22:21:58.005091 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781821065380026-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781821065380026-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781821065380026-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45999
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781821065380026-393-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44807
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45999
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:21:58.115993 20772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:21:58.116308 20772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:21:58.116391 20772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:21:58.120433 20772 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:21:58.120514 20772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:21:58.120550 20772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:21:58.120605 20772 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:21:58.120643 20772 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:21:58.125864 20772 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44807
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781821065380026-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781821065380026-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45999
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781821065380026-393-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20772
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781821065380026-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:21:58.126932 20772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:21:58.127926 20772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:21:58.136353 20778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:21:58.136359 20780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:21:58.136390 20777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:21:58.136408 20772 server_base.cc:1034] running on GCE node
I20260618 22:21:58.137085 20772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:21:58.138195 20772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:21:58.139366 20772 hybrid_clock.cc:648] HybridClock initialized: now 1781821318139330 us; error 47 us; skew 500 ppm
Jun 18 22:21:58 dist-test-slave-g6r4 krb5kdc[20759](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821318, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:21:58.142036 20772 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260618 22:21:58.142812 20772 webserver.cc:460] Webserver started at http://127.0.0.1:46269/ using document root <none> and password file <none>
I20260618 22:21:58.143299 20772 fs_manager.cc:359] Metadata directory not provided
I20260618 22:21:58.143354 20772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:21:58.143510 20772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:21:58.145380 20772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781821065380026-393-0/minicluster-data/master-0/data/instance:
uuid: "fad1e44d37a54c23abe6bd6345a56e17"
format_stamp: "Formatted at 2026-06-18 22:21:58 on dist-test-slave-g6r4"
server_key: "a498ad00b9c6f226d86f40afe5323e0e"
server_key_iv: "3242f9102aab7252c4ac1b5f9a573630"
server_key_version: "encryptionkey@0"
I20260618 22:21:58.145881 20772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781821065380026-393-0/minicluster-data/master-0/wal/instance:
uuid: "fad1e44d37a54c23abe6bd6345a56e17"
format_stamp: "Formatted at 2026-06-18 22:21:58 on dist-test-slave-g6r4"
server_key: "a498ad00b9c6f226d86f40afe5323e0e"
server_key_iv: "3242f9102aab7252c4ac1b5f9a573630"
server_key_version: "encryptionkey@0"
I20260618 22:21:58.149513 20772 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:21:58.152060 20787 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:58.153393 20772 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:21:58.153548 20772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781821065380026-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781821065380026-393-0/minicluster-data/master-0/wal
uuid: "fad1e44d37a54c23abe6bd6345a56e17"
format_stamp: "Formatted at 2026-06-18 22:21:58 on dist-test-slave-g6r4"
server_key: "a498ad00b9c6f226d86f40afe5323e0e"
server_key_iv: "3242f9102aab7252c4ac1b5f9a573630"
server_key_version: "encryptionkey@0"
I20260618 22:21:58.153668 20772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781821065380026-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781821065380026-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781821065380026-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:21:58.171718 20772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:21:58.174543 20772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:21:58.174741 20772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:21:58.181782 20772 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45999
I20260618 22:21:58.181802 20839 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45999 every 8 connection(s)
I20260618 22:21:58.182723 20772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
I20260618 22:21:58.185406 20840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:21:58.190814 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 20772
I20260618 22:21:58.190843 20840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fad1e44d37a54c23abe6bd6345a56e17: Bootstrap starting.
I20260618 22:21:58.191001 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781821065380026-393-0/minicluster-data/master-0/wal/instance
I20260618 22:21:58.191299 393 external_mini_cluster.cc:1428] Setting key 8eb2872a93ecd80cf2456a85cf181424
I20260618 22:21:58.193346 20840 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fad1e44d37a54c23abe6bd6345a56e17: Neither blocks nor log segments found. Creating new log.
I20260618 22:21:58.194152 20840 log.cc:824] T 00000000000000000000000000000000 P fad1e44d37a54c23abe6bd6345a56e17: Log is configured to *not* fsync() on all Append() calls
I20260618 22:21:58.196072 20840 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fad1e44d37a54c23abe6bd6345a56e17: No bootstrap required, opened a new log
I20260618 22:21:58.200278 20843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:58.192592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42229 (local address 127.0.0.1:45999)
0618 22:21:58.193009 (+ 417us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:58.193024 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:58.193059 (+ 35us) server_negotiation.cc:407] Connection header received
0618 22:21:58.193714 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:58.193737 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:58.194156 (+ 419us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:58.194464 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:21:58.194889 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:58.195864 (+ 975us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:58.197045 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:58.197489 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:58.197603 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:58.198513 (+ 910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:21:58.198518 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:21:58.198529 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:21:58.198955 (+ 426us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:21:58.199008 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:21:58.199030 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:21:58.199142 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:58.199385 (+ 243us) server_negotiation.cc:299] Negotiation successful
0618 22:21:58.199521 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":140,"threads_started":1}
I20260618 22:21:58.200146 20840 raft_consensus.cc:348] T 00000000000000000000000000000000 P fad1e44d37a54c23abe6bd6345a56e17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fad1e44d37a54c23abe6bd6345a56e17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45999 } }
I20260618 22:21:58.200404 20840 raft_consensus.cc:374] T 00000000000000000000000000000000 P fad1e44d37a54c23abe6bd6345a56e17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:21:58.200495 20840 raft_consensus.cc:729] T 00000000000000000000000000000000 P fad1e44d37a54c23abe6bd6345a56e17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fad1e44d37a54c23abe6bd6345a56e17, State: Initialized, Role: FOLLOWER
I20260618 22:21:58.200976 20840 consensus_queue.cc:260] T 00000000000000000000000000000000 P fad1e44d37a54c23abe6bd6345a56e17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fad1e44d37a54c23abe6bd6345a56e17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45999 } }
I20260618 22:21:58.201130 20840 raft_consensus.cc:388] T 00000000000000000000000000000000 P fad1e44d37a54c23abe6bd6345a56e17 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:21:58.201215 20840 raft_consensus.cc:482] T 00000000000000000000000000000000 P fad1e44d37a54c23abe6bd6345a56e17 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:21:58.201315 20840 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fad1e44d37a54c23abe6bd6345a56e17 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:21:58.202345 20840 raft_consensus.cc:504] T 00000000000000000000000000000000 P fad1e44d37a54c23abe6bd6345a56e17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fad1e44d37a54c23abe6bd6345a56e17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45999 } }
I20260618 22:21:58.202725 20840 leader_election.cc:302] T 00000000000000000000000000000000 P fad1e44d37a54c23abe6bd6345a56e17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fad1e44d37a54c23abe6bd6345a56e17; no voters:
I20260618 22:21:58.203115 20840 leader_election.cc:288] T 00000000000000000000000000000000 P fad1e44d37a54c23abe6bd6345a56e17 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:21:58.203210 20845 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fad1e44d37a54c23abe6bd6345a56e17 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:21:58.203231 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:21:58.203440 20845 raft_consensus.cc:686] T 00000000000000000000000000000000 P fad1e44d37a54c23abe6bd6345a56e17 [term 1 LEADER]: Becoming Leader. State: Replica: fad1e44d37a54c23abe6bd6345a56e17, State: Running, Role: LEADER
I20260618 22:21:58.203855 20845 consensus_queue.cc:237] T 00000000000000000000000000000000 P fad1e44d37a54c23abe6bd6345a56e17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fad1e44d37a54c23abe6bd6345a56e17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45999 } }
I20260618 22:21:58.204264 20840 sys_catalog.cc:564] T 00000000000000000000000000000000 P fad1e44d37a54c23abe6bd6345a56e17 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:21:58.205467 20846 sys_catalog.cc:455] T 00000000000000000000000000000000 P fad1e44d37a54c23abe6bd6345a56e17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fad1e44d37a54c23abe6bd6345a56e17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fad1e44d37a54c23abe6bd6345a56e17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45999 } } }
I20260618 22:21:58.205593 20846 sys_catalog.cc:458] T 00000000000000000000000000000000 P fad1e44d37a54c23abe6bd6345a56e17 [sys.catalog]: This master's current role is: LEADER
I20260618 22:21:58.205835 20846 sys_catalog.cc:455] T 00000000000000000000000000000000 P fad1e44d37a54c23abe6bd6345a56e17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fad1e44d37a54c23abe6bd6345a56e17. Latest consensus state: current_term: 1 leader_uuid: "fad1e44d37a54c23abe6bd6345a56e17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fad1e44d37a54c23abe6bd6345a56e17" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45999 } } }
I20260618 22:21:58.206074 20846 sys_catalog.cc:458] T 00000000000000000000000000000000 P fad1e44d37a54c23abe6bd6345a56e17 [sys.catalog]: This master's current role is: LEADER
I20260618 22:21:58.206462 20859 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:21:58.209820 20859 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:21:58.209990 20843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:58.206052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38063 (local address 127.0.0.1:45999)
0618 22:21:58.206193 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:58.206197 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:58.206219 (+ 22us) server_negotiation.cc:407] Connection header received
0618 22:21:58.206254 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:58.206260 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:58.206312 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:58.206396 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:21:58.206719 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:58.207425 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:58.208273 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:21:58.208494 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:21:58.208601 (+ 107us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:21:58.209437 (+ 836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:21:58.209440 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:21:58.209443 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:21:58.209455 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:21:58.209494 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:21:58.209505 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:21:58.209601 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:58.209723 (+ 122us) server_negotiation.cc:299] Negotiation successful
0618 22:21:58.209755 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260618 22:21:58.215265 20859 catalog_manager.cc:1347] Generated new cluster ID: c44cb3b2dabb44f4a7b420ac5940ab36
I20260618 22:21:58.215345 20859 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:21:58.236121 20859 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:21:58.237023 20859 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:21:58.241211 20859 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fad1e44d37a54c23abe6bd6345a56e17: Generated new TSK 0
I20260618 22:21:58.241994 20859 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:21:58.245285 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 20772
W20260618 22:21:58.250535 20757 connection.cc:535] client connection to 127.0.0.1:45999 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45999): Connection reset by peer (error 104)
2026-06-18T22:21:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1399 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781821065380026-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:21:58 dist-test-slave-g6r4 krb5kdc[20871](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:21:58 dist-test-slave-g6r4 krb5kdc[20871](info): set up 2 sockets
Jun 18 22:21:58 dist-test-slave-g6r4 krb5kdc[20871](info): commencing operation
krb5kdc: starting...
W20260618 22:21:59.286255 393 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781821065380026-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781821065380026-393-0/krb5kdc/test-user.keytab.
Jun 18 22:21:59 dist-test-slave-g6r4 krb5kdc[20871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:21:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:21:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781821065380026-393-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781821065380026-393-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260618 22:21:59.407295 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781821065380026-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781821065380026-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781821065380026-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43587
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781821065380026-393-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43587
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:21:59.515175 20884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:21:59.515424 20884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:21:59.515465 20884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:21:59.519377 20884 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:21:59.519434 20884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:21:59.519454 20884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:21:59.519469 20884 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:21:59.519484 20884 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:21:59.524524 20884 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781821065380026-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781821065380026-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43587
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781821065380026-393-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20884
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781821065380026-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:21:59.525424 20884 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:21:59.526417 20884 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260618 22:21:59.534605 20884 server_base.cc:1034] running on GCE node
W20260618 22:21:59.534494 20890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:21:59.534554 20892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:21:59.534494 20889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:21:59.535298 20884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:21:59.536363 20884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:21:59.537550 20884 hybrid_clock.cc:648] HybridClock initialized: now 1781821319537532 us; error 30 us; skew 500 ppm
Jun 18 22:21:59 dist-test-slave-g6r4 krb5kdc[20871](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821319, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:21:59.540287 20884 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260618 22:21:59.541101 20884 webserver.cc:460] Webserver started at http://127.0.0.1:41853/ using document root <none> and password file <none>
I20260618 22:21:59.541638 20884 fs_manager.cc:359] Metadata directory not provided
I20260618 22:21:59.541781 20884 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:21:59.542047 20884 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:21:59.544008 20884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781821065380026-393-0/minicluster-data/master-0/data/instance:
uuid: "07993c6ae86242b1b1c4dc6d3bc15b13"
format_stamp: "Formatted at 2026-06-18 22:21:59 on dist-test-slave-g6r4"
server_key: "b32430a1f8e07f01c3f7c3261660a6f9"
server_key_iv: "3978f0a48fc2b7e34b692bf896cc3952"
server_key_version: "encryptionkey@0"
I20260618 22:21:59.544567 20884 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781821065380026-393-0/minicluster-data/master-0/wal/instance:
uuid: "07993c6ae86242b1b1c4dc6d3bc15b13"
format_stamp: "Formatted at 2026-06-18 22:21:59 on dist-test-slave-g6r4"
server_key: "b32430a1f8e07f01c3f7c3261660a6f9"
server_key_iv: "3978f0a48fc2b7e34b692bf896cc3952"
server_key_version: "encryptionkey@0"
I20260618 22:21:59.548194 20884 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260618 22:21:59.550513 20899 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:21:59.551632 20884 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:21:59.551772 20884 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781821065380026-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781821065380026-393-0/minicluster-data/master-0/wal
uuid: "07993c6ae86242b1b1c4dc6d3bc15b13"
format_stamp: "Formatted at 2026-06-18 22:21:59 on dist-test-slave-g6r4"
server_key: "b32430a1f8e07f01c3f7c3261660a6f9"
server_key_iv: "3978f0a48fc2b7e34b692bf896cc3952"
server_key_version: "encryptionkey@0"
I20260618 22:21:59.551875 20884 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781821065380026-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781821065380026-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781821065380026-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:21:59.556269 20884 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:21:59.559342 20884 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:21:59.559525 20884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:21:59.566030 20884 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43587
I20260618 22:21:59.566059 20951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43587 every 8 connection(s)
I20260618 22:21:59.566900 20884 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
I20260618 22:21:59.569471 20952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:21:59.572940 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 20884
I20260618 22:21:59.573089 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781821065380026-393-0/minicluster-data/master-0/wal/instance
I20260618 22:21:59.573376 393 external_mini_cluster.cc:1428] Setting key 990e1a8bd2ca552be9dde90c3c4a8cd3
I20260618 22:21:59.574946 20952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07993c6ae86242b1b1c4dc6d3bc15b13: Bootstrap starting.
I20260618 22:21:59.577469 20952 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 07993c6ae86242b1b1c4dc6d3bc15b13: Neither blocks nor log segments found. Creating new log.
I20260618 22:21:59.578353 20952 log.cc:824] T 00000000000000000000000000000000 P 07993c6ae86242b1b1c4dc6d3bc15b13: Log is configured to *not* fsync() on all Append() calls
I20260618 22:21:59.578778 20955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:59.574718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34663 (local address 127.0.0.1:43587)
0618 22:21:59.575118 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:59.575135 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:59.575177 (+ 42us) server_negotiation.cc:407] Connection header received
0618 22:21:59.575948 (+ 771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:59.575973 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:59.576328 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:59.576656 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:21:59.577329 (+ 673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:21:59.577336 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:21:59.577351 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:21:59.577366 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:21:59.577428 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:21:59.577458 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:21:59.577571 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:59.577738 (+ 167us) server_negotiation.cc:299] Negotiation successful
0618 22:21:59.577881 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":103,"threads_started":1}
I20260618 22:21:59.580264 20952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 07993c6ae86242b1b1c4dc6d3bc15b13: No bootstrap required, opened a new log
I20260618 22:21:59.581447 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:21:59.584641 20952 raft_consensus.cc:348] T 00000000000000000000000000000000 P 07993c6ae86242b1b1c4dc6d3bc15b13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07993c6ae86242b1b1c4dc6d3bc15b13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43587 } }
I20260618 22:21:59.584903 20952 raft_consensus.cc:374] T 00000000000000000000000000000000 P 07993c6ae86242b1b1c4dc6d3bc15b13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:21:59.585009 20952 raft_consensus.cc:729] T 00000000000000000000000000000000 P 07993c6ae86242b1b1c4dc6d3bc15b13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 07993c6ae86242b1b1c4dc6d3bc15b13, State: Initialized, Role: FOLLOWER
I20260618 22:21:59.585040 20955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:21:59.583615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55857 (local address 127.0.0.1:43587)
0618 22:21:59.583748 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0618 22:21:59.583755 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:21:59.583828 (+ 73us) server_negotiation.cc:407] Connection header received
0618 22:21:59.583931 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:21:59.583933 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:21:59.583962 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:21:59.584037 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:21:59.584655 (+ 618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:21:59.584661 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:21:59.584665 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:21:59.584676 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:21:59.584701 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:21:59.584713 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:21:59.584808 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0618 22:21:59.584866 (+ 58us) server_negotiation.cc:299] Negotiation successful
0618 22:21:59.584899 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260618 22:21:59.585470 20952 consensus_queue.cc:260] T 00000000000000000000000000000000 P 07993c6ae86242b1b1c4dc6d3bc15b13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07993c6ae86242b1b1c4dc6d3bc15b13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43587 } }
I20260618 22:21:59.585589 20952 raft_consensus.cc:388] T 00000000000000000000000000000000 P 07993c6ae86242b1b1c4dc6d3bc15b13 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:21:59.585670 20952 raft_consensus.cc:482] T 00000000000000000000000000000000 P 07993c6ae86242b1b1c4dc6d3bc15b13 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:21:59.585772 20952 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 07993c6ae86242b1b1c4dc6d3bc15b13 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:21:59.586606 20952 raft_consensus.cc:504] T 00000000000000000000000000000000 P 07993c6ae86242b1b1c4dc6d3bc15b13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07993c6ae86242b1b1c4dc6d3bc15b13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43587 } }
I20260618 22:21:59.586866 20952 leader_election.cc:302] T 00000000000000000000000000000000 P 07993c6ae86242b1b1c4dc6d3bc15b13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 07993c6ae86242b1b1c4dc6d3bc15b13; no voters:
I20260618 22:21:59.587123 20952 leader_election.cc:288] T 00000000000000000000000000000000 P 07993c6ae86242b1b1c4dc6d3bc15b13 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:21:59.587260 20962 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 07993c6ae86242b1b1c4dc6d3bc15b13 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:21:59.587468 20962 raft_consensus.cc:686] T 00000000000000000000000000000000 P 07993c6ae86242b1b1c4dc6d3bc15b13 [term 1 LEADER]: Becoming Leader. State: Replica: 07993c6ae86242b1b1c4dc6d3bc15b13, State: Running, Role: LEADER
I20260618 22:21:59.587831 20962 consensus_queue.cc:237] T 00000000000000000000000000000000 P 07993c6ae86242b1b1c4dc6d3bc15b13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07993c6ae86242b1b1c4dc6d3bc15b13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43587 } }
I20260618 22:21:59.588280 20952 sys_catalog.cc:564] T 00000000000000000000000000000000 P 07993c6ae86242b1b1c4dc6d3bc15b13 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:21:59.589404 20963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07993c6ae86242b1b1c4dc6d3bc15b13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "07993c6ae86242b1b1c4dc6d3bc15b13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07993c6ae86242b1b1c4dc6d3bc15b13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43587 } } }
I20260618 22:21:59.589545 20963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07993c6ae86242b1b1c4dc6d3bc15b13 [sys.catalog]: This master's current role is: LEADER
I20260618 22:21:59.589964 20971 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:21:59.590623 20964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 07993c6ae86242b1b1c4dc6d3bc15b13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 07993c6ae86242b1b1c4dc6d3bc15b13. Latest consensus state: current_term: 1 leader_uuid: "07993c6ae86242b1b1c4dc6d3bc15b13" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "07993c6ae86242b1b1c4dc6d3bc15b13" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43587 } } }
I20260618 22:21:59.590720 20964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 07993c6ae86242b1b1c4dc6d3bc15b13 [sys.catalog]: This master's current role is: LEADER
I20260618 22:21:59.592521 20971 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:21:59.597883 20971 catalog_manager.cc:1347] Generated new cluster ID: e87c8757343945a981e207c88999b0a5
I20260618 22:21:59.597985 20971 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:21:59.612735 20971 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:21:59.613087 20971 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:21:59.622589 20971 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 07993c6ae86242b1b1c4dc6d3bc15b13: Generated new TSK 0
I20260618 22:21:59.623272 20971 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:21:59.632354 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 20884
2026-06-18T22:21:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1386 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781821065380026-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:21:59 dist-test-slave-g6r4 krb5kdc[20983](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:21:59 dist-test-slave-g6r4 krb5kdc[20983](info): set up 2 sockets
Jun 18 22:21:59 dist-test-slave-g6r4 krb5kdc[20983](info): commencing operation
krb5kdc: starting...
W20260618 22:22:00.678121 393 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781821065380026-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781821065380026-393-0/krb5kdc/test-user.keytab.
Jun 18 22:22:00 dist-test-slave-g6r4 krb5kdc[20983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:22:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:22:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781821065380026-393-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781821065380026-393-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260618 22:22:00.796191 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781821065380026-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781821065380026-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781821065380026-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39181
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781821065380026-393-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39181
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:22:00.902091 20996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:22:00.902371 20996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:22:00.902436 20996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:22:00.908461 20996 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:22:00.908520 20996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:22:00.908540 20996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:22:00.908556 20996 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:22:00.908571 20996 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:22:00.913789 20996 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781821065380026-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781821065380026-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781821065380026-393-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781821065380026-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:22:00.914736 20996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:22:00.915629 20996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:22:00.923652 21001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:22:00.923760 21004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:22:00.923863 20996 server_base.cc:1034] running on GCE node
W20260618 22:22:00.923982 21002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:22:00.924523 20996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:22:00.925586 20996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:22:00.926752 20996 hybrid_clock.cc:648] HybridClock initialized: now 1781821320926721 us; error 44 us; skew 500 ppm
Jun 18 22:22:00 dist-test-slave-g6r4 krb5kdc[20983](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821320, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:22:00.934021 20996 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260618 22:22:00.934872 20996 webserver.cc:460] Webserver started at http://127.0.0.1:46013/ using document root <none> and password file <none>
I20260618 22:22:00.935411 20996 fs_manager.cc:359] Metadata directory not provided
I20260618 22:22:00.935472 20996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:22:00.935734 20996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:22:00.937719 20996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781821065380026-393-0/minicluster-data/master-0/data/instance:
uuid: "a1dd56da9a844ecc987cf3fc5b4a688f"
format_stamp: "Formatted at 2026-06-18 22:22:00 on dist-test-slave-g6r4"
server_key: "823a6e232dbc4ec9948d15cb1cde9c55"
server_key_iv: "35a18a7469f7623847145dc07d5d7fe8"
server_key_version: "encryptionkey@0"
I20260618 22:22:00.938326 20996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781821065380026-393-0/minicluster-data/master-0/wal/instance:
uuid: "a1dd56da9a844ecc987cf3fc5b4a688f"
format_stamp: "Formatted at 2026-06-18 22:22:00 on dist-test-slave-g6r4"
server_key: "823a6e232dbc4ec9948d15cb1cde9c55"
server_key_iv: "35a18a7469f7623847145dc07d5d7fe8"
server_key_version: "encryptionkey@0"
I20260618 22:22:00.941825 20996 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:22:00.944099 21011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:22:00.945304 20996 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:22:00.945475 20996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781821065380026-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781821065380026-393-0/minicluster-data/master-0/wal
uuid: "a1dd56da9a844ecc987cf3fc5b4a688f"
format_stamp: "Formatted at 2026-06-18 22:22:00 on dist-test-slave-g6r4"
server_key: "823a6e232dbc4ec9948d15cb1cde9c55"
server_key_iv: "35a18a7469f7623847145dc07d5d7fe8"
server_key_version: "encryptionkey@0"
I20260618 22:22:00.945588 20996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781821065380026-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781821065380026-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781821065380026-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:22:00.974314 20996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:22:00.977103 20996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:22:00.977322 20996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:22:00.984076 20996 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39181
I20260618 22:22:00.984095 21063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39181 every 8 connection(s)
I20260618 22:22:00.984965 20996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
I20260618 22:22:00.987660 21064 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:22:00.991869 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 20996
I20260618 22:22:00.992023 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781821065380026-393-0/minicluster-data/master-0/wal/instance
I20260618 22:22:00.992322 393 external_mini_cluster.cc:1428] Setting key a8104409079664e3bea73fe136f4b67f
I20260618 22:22:00.993258 21064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1dd56da9a844ecc987cf3fc5b4a688f: Bootstrap starting.
I20260618 22:22:00.995482 21064 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1dd56da9a844ecc987cf3fc5b4a688f: Neither blocks nor log segments found. Creating new log.
I20260618 22:22:00.996372 21064 log.cc:824] T 00000000000000000000000000000000 P a1dd56da9a844ecc987cf3fc5b4a688f: Log is configured to *not* fsync() on all Append() calls
I20260618 22:22:00.998471 21064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1dd56da9a844ecc987cf3fc5b4a688f: No bootstrap required, opened a new log
Jun 18 22:22:00 dist-test-slave-g6r4 krb5kdc[20983](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821320, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260618 22:22:01.002653 21064 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1dd56da9a844ecc987cf3fc5b4a688f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1dd56da9a844ecc987cf3fc5b4a688f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39181 } }
I20260618 22:22:01.002914 21064 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1dd56da9a844ecc987cf3fc5b4a688f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:22:01.003001 21064 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1dd56da9a844ecc987cf3fc5b4a688f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1dd56da9a844ecc987cf3fc5b4a688f, State: Initialized, Role: FOLLOWER
I20260618 22:22:01.003461 21064 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1dd56da9a844ecc987cf3fc5b4a688f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1dd56da9a844ecc987cf3fc5b4a688f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39181 } }
I20260618 22:22:01.003609 21064 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1dd56da9a844ecc987cf3fc5b4a688f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:22:01.003700 21064 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1dd56da9a844ecc987cf3fc5b4a688f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:22:01.003795 21064 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1dd56da9a844ecc987cf3fc5b4a688f [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:22:01.004601 21064 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1dd56da9a844ecc987cf3fc5b4a688f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1dd56da9a844ecc987cf3fc5b4a688f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39181 } }
I20260618 22:22:01.004923 21064 leader_election.cc:302] T 00000000000000000000000000000000 P a1dd56da9a844ecc987cf3fc5b4a688f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1dd56da9a844ecc987cf3fc5b4a688f; no voters:
I20260618 22:22:01.005231 21064 leader_election.cc:288] T 00000000000000000000000000000000 P a1dd56da9a844ecc987cf3fc5b4a688f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:22:01.005293 21069 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1dd56da9a844ecc987cf3fc5b4a688f [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:22:01.005507 21069 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1dd56da9a844ecc987cf3fc5b4a688f [term 1 LEADER]: Becoming Leader. State: Replica: a1dd56da9a844ecc987cf3fc5b4a688f, State: Running, Role: LEADER
I20260618 22:22:01.005836 21069 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1dd56da9a844ecc987cf3fc5b4a688f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1dd56da9a844ecc987cf3fc5b4a688f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39181 } }
I20260618 22:22:01.006510 21064 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1dd56da9a844ecc987cf3fc5b4a688f [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:22:01.006752 21067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:00.993600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55129 (local address 127.0.0.1:39181)
0618 22:22:00.993994 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:00.994010 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:00.994044 (+ 34us) server_negotiation.cc:407] Connection header received
0618 22:22:00.994646 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:00.994662 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:00.994980 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:00.995244 (+ 264us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:00.996164 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:00.997364 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:00.998594 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:00.999074 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:00.999193 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:01.001959 (+ 2766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:22:01.001973 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:22:01.001986 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:22:01.002012 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:22:01.003860 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:01.004283 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:01.004288 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:01.004294 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:01.004427 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:01.004701 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:01.004706 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:01.004707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:01.005142 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:22:01.005356 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0618 22:22:01.005608 (+ 252us) server_negotiation.cc:299] Negotiation successful
0618 22:22:01.005940 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":124,"threads_started":1}
I20260618 22:22:01.007004 21069 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1dd56da9a844ecc987cf3fc5b4a688f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1dd56da9a844ecc987cf3fc5b4a688f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1dd56da9a844ecc987cf3fc5b4a688f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39181 } } }
I20260618 22:22:01.007135 21069 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1dd56da9a844ecc987cf3fc5b4a688f [sys.catalog]: This master's current role is: LEADER
I20260618 22:22:01.007599 21075 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:22:01.008409 21071 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1dd56da9a844ecc987cf3fc5b4a688f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1dd56da9a844ecc987cf3fc5b4a688f. Latest consensus state: current_term: 1 leader_uuid: "a1dd56da9a844ecc987cf3fc5b4a688f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1dd56da9a844ecc987cf3fc5b4a688f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39181 } } }
I20260618 22:22:01.008562 21071 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1dd56da9a844ecc987cf3fc5b4a688f [sys.catalog]: This master's current role is: LEADER
I20260618 22:22:01.009868 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:22:01.010828 21075 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:22:01.016991 21075 catalog_manager.cc:1347] Generated new cluster ID: ca59a87480dd4dc9a4d8b51be70bc5bc
I20260618 22:22:01.017096 21075 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:22:01.020372 21067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:01.012018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50187 (local address 127.0.0.1:39181)
0618 22:22:01.012112 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:01.012116 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:01.012297 (+ 181us) server_negotiation.cc:407] Connection header received
0618 22:22:01.012389 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:01.012392 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:01.012437 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:01.012521 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:01.013299 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:01.014121 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:01.015172 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:01.015551 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:01.015651 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:01.017087 (+ 1436us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:22:01.017090 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:22:01.017093 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:22:01.017118 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:22:01.018756 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:01.019232 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:01.019235 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:01.019237 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:01.019311 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:01.019594 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:01.019597 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:01.019598 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:01.019767 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:22:01.019878 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0618 22:22:01.020123 (+ 245us) server_negotiation.cc:299] Negotiation successful
0618 22:22:01.020228 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260618 22:22:01.048422 21075 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:22:01.049471 21075 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:22:01.059864 21075 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1dd56da9a844ecc987cf3fc5b4a688f: Generated new TSK 0
I20260618 22:22:01.060602 21075 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:22:01.072403 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 20996
2026-06-18T22:22:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1439 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781821065380026-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:22:01 dist-test-slave-g6r4 krb5kdc[21095](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:22:01 dist-test-slave-g6r4 krb5kdc[21095](info): set up 2 sockets
Jun 18 22:22:01 dist-test-slave-g6r4 krb5kdc[21095](info): commencing operation
krb5kdc: starting...
W20260618 22:22:02.114838 393 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781821065380026-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781821065380026-393-0/krb5kdc/test-user.keytab.
Jun 18 22:22:02 dist-test-slave-g6r4 krb5kdc[21095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:22:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:22:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781821065380026-393-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781821065380026-393-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260618 22:22:02.230655 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781821065380026-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781821065380026-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781821065380026-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38085
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781821065380026-393-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35195
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38085
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:22:02.335283 21108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:22:02.335513 21108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:22:02.335552 21108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:22:02.339450 21108 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:22:02.339497 21108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:22:02.339516 21108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:22:02.339532 21108 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:22:02.339587 21108 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:22:02.344554 21108 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35195
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781821065380026-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781821065380026-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38085
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781821065380026-393-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781821065380026-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:22:02.345451 21108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:22:02.346390 21108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:22:02.354468 21113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:22:02.354463 21114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:22:02.354547 21116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:22:02.354668 21108 server_base.cc:1034] running on GCE node
I20260618 22:22:02.355410 21108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:22:02.356384 21108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:22:02.357560 21108 hybrid_clock.cc:648] HybridClock initialized: now 1781821322357525 us; error 48 us; skew 500 ppm
Jun 18 22:22:02 dist-test-slave-g6r4 krb5kdc[21095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821322, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:22:02.360847 21108 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260618 22:22:02.361819 21108 webserver.cc:460] Webserver started at http://127.0.0.1:38333/ using document root <none> and password file <none>
I20260618 22:22:02.362634 21108 fs_manager.cc:359] Metadata directory not provided
I20260618 22:22:02.362725 21108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:22:02.362973 21108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:22:02.364959 21108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781821065380026-393-0/minicluster-data/master-0/data/instance:
uuid: "f6bb1f31f2cf42ab9d65455634cf20bd"
format_stamp: "Formatted at 2026-06-18 22:22:02 on dist-test-slave-g6r4"
server_key: "a1b89873daa4d4a00aa0ffe71005f418"
server_key_iv: "b18e623977a7700d227c88a8931f388f"
server_key_version: "encryptionkey@0"
I20260618 22:22:02.365520 21108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781821065380026-393-0/minicluster-data/master-0/wal/instance:
uuid: "f6bb1f31f2cf42ab9d65455634cf20bd"
format_stamp: "Formatted at 2026-06-18 22:22:02 on dist-test-slave-g6r4"
server_key: "a1b89873daa4d4a00aa0ffe71005f418"
server_key_iv: "b18e623977a7700d227c88a8931f388f"
server_key_version: "encryptionkey@0"
I20260618 22:22:02.369065 21108 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260618 22:22:02.371380 21123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:22:02.372591 21108 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260618 22:22:02.372757 21108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781821065380026-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781821065380026-393-0/minicluster-data/master-0/wal
uuid: "f6bb1f31f2cf42ab9d65455634cf20bd"
format_stamp: "Formatted at 2026-06-18 22:22:02 on dist-test-slave-g6r4"
server_key: "a1b89873daa4d4a00aa0ffe71005f418"
server_key_iv: "b18e623977a7700d227c88a8931f388f"
server_key_version: "encryptionkey@0"
I20260618 22:22:02.372867 21108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781821065380026-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781821065380026-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781821065380026-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:22:02.385519 21108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:22:02.388479 21108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:22:02.388700 21108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:22:02.396170 21108 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38085
I20260618 22:22:02.396194 21175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38085 every 8 connection(s)
I20260618 22:22:02.397078 21108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
I20260618 22:22:02.399785 21176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:22:02.405128 21176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6bb1f31f2cf42ab9d65455634cf20bd: Bootstrap starting.
I20260618 22:22:02.406059 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 21108
I20260618 22:22:02.406203 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781821065380026-393-0/minicluster-data/master-0/wal/instance
I20260618 22:22:02.406540 393 external_mini_cluster.cc:1428] Setting key 8b92b259f08efe8a208ad5cd3a2fde32
I20260618 22:22:02.408072 21176 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f6bb1f31f2cf42ab9d65455634cf20bd: Neither blocks nor log segments found. Creating new log.
I20260618 22:22:02.408793 21176 log.cc:824] T 00000000000000000000000000000000 P f6bb1f31f2cf42ab9d65455634cf20bd: Log is configured to *not* fsync() on all Append() calls
I20260618 22:22:02.410844 21176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f6bb1f31f2cf42ab9d65455634cf20bd: No bootstrap required, opened a new log
I20260618 22:22:02.414422 21176 raft_consensus.cc:348] T 00000000000000000000000000000000 P f6bb1f31f2cf42ab9d65455634cf20bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6bb1f31f2cf42ab9d65455634cf20bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38085 } }
I20260618 22:22:02.414723 21176 raft_consensus.cc:374] T 00000000000000000000000000000000 P f6bb1f31f2cf42ab9d65455634cf20bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:22:02.414803 21176 raft_consensus.cc:729] T 00000000000000000000000000000000 P f6bb1f31f2cf42ab9d65455634cf20bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f6bb1f31f2cf42ab9d65455634cf20bd, State: Initialized, Role: FOLLOWER
I20260618 22:22:02.415130 21179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:02.407614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52965 (local address 127.0.0.1:38085)
0618 22:22:02.408013 (+ 399us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:02.408037 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:02.408071 (+ 34us) server_negotiation.cc:407] Connection header received
0618 22:22:02.408722 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:02.408738 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:02.409141 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:02.409537 (+ 396us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:02.409858 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:02.410861 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:02.411800 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:02.412263 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:02.412446 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:02.413290 (+ 844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:22:02.413296 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:22:02.413308 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:22:02.413693 (+ 385us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:22:02.413744 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:22:02.413766 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:22:02.413878 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0618 22:22:02.414034 (+ 156us) server_negotiation.cc:299] Negotiation successful
0618 22:22:02.414242 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":171,"threads_started":1}
I20260618 22:22:02.415292 21176 consensus_queue.cc:260] T 00000000000000000000000000000000 P f6bb1f31f2cf42ab9d65455634cf20bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6bb1f31f2cf42ab9d65455634cf20bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38085 } }
I20260618 22:22:02.415452 21176 raft_consensus.cc:388] T 00000000000000000000000000000000 P f6bb1f31f2cf42ab9d65455634cf20bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:22:02.415539 21176 raft_consensus.cc:482] T 00000000000000000000000000000000 P f6bb1f31f2cf42ab9d65455634cf20bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:22:02.415637 21176 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f6bb1f31f2cf42ab9d65455634cf20bd [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:22:02.416538 21176 raft_consensus.cc:504] T 00000000000000000000000000000000 P f6bb1f31f2cf42ab9d65455634cf20bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6bb1f31f2cf42ab9d65455634cf20bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38085 } }
I20260618 22:22:02.416908 21176 leader_election.cc:302] T 00000000000000000000000000000000 P f6bb1f31f2cf42ab9d65455634cf20bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f6bb1f31f2cf42ab9d65455634cf20bd; no voters:
I20260618 22:22:02.417325 21176 leader_election.cc:288] T 00000000000000000000000000000000 P f6bb1f31f2cf42ab9d65455634cf20bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:22:02.417412 21181 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f6bb1f31f2cf42ab9d65455634cf20bd [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:22:02.417656 21181 raft_consensus.cc:686] T 00000000000000000000000000000000 P f6bb1f31f2cf42ab9d65455634cf20bd [term 1 LEADER]: Becoming Leader. State: Replica: f6bb1f31f2cf42ab9d65455634cf20bd, State: Running, Role: LEADER
I20260618 22:22:02.418079 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:22:02.418038 21181 consensus_queue.cc:237] T 00000000000000000000000000000000 P f6bb1f31f2cf42ab9d65455634cf20bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6bb1f31f2cf42ab9d65455634cf20bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38085 } }
I20260618 22:22:02.418678 21176 sys_catalog.cc:564] T 00000000000000000000000000000000 P f6bb1f31f2cf42ab9d65455634cf20bd [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:22:02.420603 21182 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6bb1f31f2cf42ab9d65455634cf20bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f6bb1f31f2cf42ab9d65455634cf20bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6bb1f31f2cf42ab9d65455634cf20bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38085 } } }
I20260618 22:22:02.420784 21182 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6bb1f31f2cf42ab9d65455634cf20bd [sys.catalog]: This master's current role is: LEADER
I20260618 22:22:02.420779 21183 sys_catalog.cc:455] T 00000000000000000000000000000000 P f6bb1f31f2cf42ab9d65455634cf20bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader f6bb1f31f2cf42ab9d65455634cf20bd. Latest consensus state: current_term: 1 leader_uuid: "f6bb1f31f2cf42ab9d65455634cf20bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f6bb1f31f2cf42ab9d65455634cf20bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38085 } } }
I20260618 22:22:02.420933 21183 sys_catalog.cc:458] T 00000000000000000000000000000000 P f6bb1f31f2cf42ab9d65455634cf20bd [sys.catalog]: This master's current role is: LEADER
W20260618 22:22:02.424069 21200 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f6bb1f31f2cf42ab9d65455634cf20bd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 22:22:02.424150 21200 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 22:22:02.424209 21201 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:22:02.425091 21201 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:22:02.425513 21179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:02.421340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44955 (local address 127.0.0.1:38085)
0618 22:22:02.421449 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:02.421454 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:02.421952 (+ 498us) server_negotiation.cc:407] Connection header received
0618 22:22:02.422134 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:02.422137 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:02.422184 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:02.422268 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:02.422546 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:02.423064 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:02.423881 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:02.424107 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:02.424223 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:02.425078 (+ 855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:22:02.425082 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:22:02.425085 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:22:02.425096 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:22:02.425128 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:22:02.425139 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:22:02.425215 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0618 22:22:02.425306 (+ 91us) server_negotiation.cc:299] Negotiation successful
0618 22:22:02.425361 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260618 22:22:02.430739 21201 catalog_manager.cc:1347] Generated new cluster ID: da1d3fb1471f4172a51eff831ff0d643
I20260618 22:22:02.430800 21201 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:22:02.462510 21201 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:22:02.463474 21201 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:22:02.470364 21201 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f6bb1f31f2cf42ab9d65455634cf20bd: Generated new TSK 0
I20260618 22:22:02.471024 21201 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:22:02.479851 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 21108
2026-06-18T22:22:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1407 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781821065380026-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:22:02 dist-test-slave-g6r4 krb5kdc[21207](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:22:02 dist-test-slave-g6r4 krb5kdc[21207](info): set up 2 sockets
Jun 18 22:22:02 dist-test-slave-g6r4 krb5kdc[21207](info): commencing operation
krb5kdc: starting...
W20260618 22:22:03.522117 393 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781821065380026-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781821065380026-393-0/krb5kdc/test-user.keytab.
Jun 18 22:22:03 dist-test-slave-g6r4 krb5kdc[21207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821323, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:22:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:22:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781821065380026-393-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781821065380026-393-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260618 22:22:03.640457 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781821065380026-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781821065380026-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781821065380026-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41323
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781821065380026-393-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41323
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:22:03.746169 21220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:22:03.746402 21220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:22:03.746441 21220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:22:03.750448 21220 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:22:03.750497 21220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:22:03.750517 21220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:22:03.750533 21220 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:22:03.750550 21220 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:22:03.756660 21220 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781821065380026-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781821065380026-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41323
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781821065380026-393-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41323
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21220
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781821065380026-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:22:03.757620 21220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:22:03.758623 21220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:22:03.766408 21228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:22:03.766403 21226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:22:03.766402 21225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:22:03.766788 21220 server_base.cc:1034] running on GCE node
I20260618 22:22:03.767292 21220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:22:03.768344 21220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:22:03.769552 21220 hybrid_clock.cc:648] HybridClock initialized: now 1781821323769529 us; error 47 us; skew 500 ppm
Jun 18 22:22:03 dist-test-slave-g6r4 krb5kdc[21207](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821323, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:22:03.772428 21220 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260618 22:22:03.773388 21220 webserver.cc:460] Webserver started at http://127.0.0.1:42967/ using document root <none> and password file <none>
I20260618 22:22:03.773998 21220 fs_manager.cc:359] Metadata directory not provided
I20260618 22:22:03.774091 21220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:22:03.774322 21220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:22:03.776410 21220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781821065380026-393-0/minicluster-data/master-0/data/instance:
uuid: "ad44df885fc244be8ff92083d6df042e"
format_stamp: "Formatted at 2026-06-18 22:22:03 on dist-test-slave-g6r4"
server_key: "7f3f08edfa65487be888abf14761fe28"
server_key_iv: "3fcd455cfaae2cc07cfca9acb1a24193"
server_key_version: "encryptionkey@0"
I20260618 22:22:03.776973 21220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781821065380026-393-0/minicluster-data/master-0/wal/instance:
uuid: "ad44df885fc244be8ff92083d6df042e"
format_stamp: "Formatted at 2026-06-18 22:22:03 on dist-test-slave-g6r4"
server_key: "7f3f08edfa65487be888abf14761fe28"
server_key_iv: "3fcd455cfaae2cc07cfca9acb1a24193"
server_key_version: "encryptionkey@0"
I20260618 22:22:03.780579 21220 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260618 22:22:03.783054 21235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:22:03.784263 21220 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260618 22:22:03.784421 21220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781821065380026-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781821065380026-393-0/minicluster-data/master-0/wal
uuid: "ad44df885fc244be8ff92083d6df042e"
format_stamp: "Formatted at 2026-06-18 22:22:03 on dist-test-slave-g6r4"
server_key: "7f3f08edfa65487be888abf14761fe28"
server_key_iv: "3fcd455cfaae2cc07cfca9acb1a24193"
server_key_version: "encryptionkey@0"
I20260618 22:22:03.784536 21220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781821065380026-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781821065380026-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781821065380026-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:22:03.789057 21220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:22:03.792301 21220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:22:03.792497 21220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:22:03.799432 21220 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41323
I20260618 22:22:03.799459 21287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41323 every 8 connection(s)
I20260618 22:22:03.800282 21220 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
I20260618 22:22:03.802867 21288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:22:03.805799 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 21220
I20260618 22:22:03.805968 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781821065380026-393-0/minicluster-data/master-0/wal/instance
I20260618 22:22:03.806237 393 external_mini_cluster.cc:1428] Setting key 551522c7d04f6251c2a281db6d4bd402
I20260618 22:22:03.808943 21288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad44df885fc244be8ff92083d6df042e: Bootstrap starting.
I20260618 22:22:03.811025 21291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:03.807525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54807 (local address 127.0.0.1:41323)
0618 22:22:03.807949 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:03.807961 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:03.807993 (+ 32us) server_negotiation.cc:407] Connection header received
0618 22:22:03.808547 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:03.808563 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:03.808861 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:03.809176 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:03.809924 (+ 748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:22:03.809929 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:22:03.809943 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:22:03.809954 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:22:03.810008 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:22:03.810027 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:22:03.810161 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0618 22:22:03.810255 (+ 94us) server_negotiation.cc:299] Negotiation successful
0618 22:22:03.810393 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":132,"threads_started":1}
I20260618 22:22:03.811439 21288 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad44df885fc244be8ff92083d6df042e: Neither blocks nor log segments found. Creating new log.
I20260618 22:22:03.812289 21288 log.cc:824] T 00000000000000000000000000000000 P ad44df885fc244be8ff92083d6df042e: Log is configured to *not* fsync() on all Append() calls
I20260618 22:22:03.814038 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:22:03.814433 21288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad44df885fc244be8ff92083d6df042e: No bootstrap required, opened a new log
I20260618 22:22:03.817512 21291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:03.816050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40457 (local address 127.0.0.1:41323)
0618 22:22:03.816160 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:03.816166 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:03.816300 (+ 134us) server_negotiation.cc:407] Connection header received
0618 22:22:03.816403 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:03.816405 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:03.816431 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:03.816496 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:03.817120 (+ 624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:22:03.817123 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:22:03.817125 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:22:03.817134 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:22:03.817156 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:22:03.817165 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:22:03.817231 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0618 22:22:03.817347 (+ 116us) server_negotiation.cc:299] Negotiation successful
0618 22:22:03.817377 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260618 22:22:03.818666 21288 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad44df885fc244be8ff92083d6df042e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad44df885fc244be8ff92083d6df042e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41323 } }
I20260618 22:22:03.818917 21288 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad44df885fc244be8ff92083d6df042e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:22:03.818980 21288 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad44df885fc244be8ff92083d6df042e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad44df885fc244be8ff92083d6df042e, State: Initialized, Role: FOLLOWER
I20260618 22:22:03.819357 21288 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad44df885fc244be8ff92083d6df042e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad44df885fc244be8ff92083d6df042e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41323 } }
I20260618 22:22:03.819499 21288 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad44df885fc244be8ff92083d6df042e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:22:03.819586 21288 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad44df885fc244be8ff92083d6df042e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:22:03.819681 21288 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad44df885fc244be8ff92083d6df042e [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:22:03.820508 21288 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad44df885fc244be8ff92083d6df042e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad44df885fc244be8ff92083d6df042e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41323 } }
I20260618 22:22:03.820832 21288 leader_election.cc:302] T 00000000000000000000000000000000 P ad44df885fc244be8ff92083d6df042e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad44df885fc244be8ff92083d6df042e; no voters:
I20260618 22:22:03.821147 21288 leader_election.cc:288] T 00000000000000000000000000000000 P ad44df885fc244be8ff92083d6df042e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:22:03.821233 21298 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad44df885fc244be8ff92083d6df042e [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:22:03.821476 21298 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad44df885fc244be8ff92083d6df042e [term 1 LEADER]: Becoming Leader. State: Replica: ad44df885fc244be8ff92083d6df042e, State: Running, Role: LEADER
I20260618 22:22:03.821825 21298 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad44df885fc244be8ff92083d6df042e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad44df885fc244be8ff92083d6df042e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41323 } }
I20260618 22:22:03.822317 21288 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad44df885fc244be8ff92083d6df042e [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:22:03.823663 21299 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad44df885fc244be8ff92083d6df042e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad44df885fc244be8ff92083d6df042e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad44df885fc244be8ff92083d6df042e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41323 } } }
I20260618 22:22:03.823779 21299 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad44df885fc244be8ff92083d6df042e [sys.catalog]: This master's current role is: LEADER
I20260618 22:22:03.823791 21300 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad44df885fc244be8ff92083d6df042e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad44df885fc244be8ff92083d6df042e. Latest consensus state: current_term: 1 leader_uuid: "ad44df885fc244be8ff92083d6df042e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad44df885fc244be8ff92083d6df042e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41323 } } }
I20260618 22:22:03.823933 21300 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad44df885fc244be8ff92083d6df042e [sys.catalog]: This master's current role is: LEADER
I20260618 22:22:03.824935 21307 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:22:03.827921 21307 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:22:03.833062 21307 catalog_manager.cc:1347] Generated new cluster ID: 1fb951265eb047ebbc1dd9ecd4bb664f
I20260618 22:22:03.833120 21307 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:22:03.844133 21307 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:22:03.844417 21307 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:22:03.854300 21307 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad44df885fc244be8ff92083d6df042e: Generated new TSK 0
I20260618 22:22:03.855108 21307 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:22:03.864413 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 21220
2026-06-18T22:22:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1384 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781821065380026-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:22:03 dist-test-slave-g6r4 krb5kdc[21319](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:22:03 dist-test-slave-g6r4 krb5kdc[21319](info): set up 2 sockets
Jun 18 22:22:03 dist-test-slave-g6r4 krb5kdc[21319](info): commencing operation
krb5kdc: starting...
W20260618 22:22:04.898350 393 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781821065380026-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781821065380026-393-0/krb5kdc/test-user.keytab.
Jun 18 22:22:04 dist-test-slave-g6r4 krb5kdc[21319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:22:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:22:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781821065380026-393-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781821065380026-393-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260618 22:22:05.016337 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781821065380026-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781821065380026-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781821065380026-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46477
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781821065380026-393-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46477
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:22:05.121956 21332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:22:05.122181 21332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:22:05.122220 21332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:22:05.126055 21332 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:22:05.126101 21332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:22:05.126127 21332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:22:05.126143 21332 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:22:05.126164 21332 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:22:05.131109 21332 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781821065380026-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781821065380026-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46477
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781821065380026-393-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781821065380026-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:22:05.131928 21332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:22:05.132776 21332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:22:05.140614 21340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:22:05.140774 21332 server_base.cc:1034] running on GCE node
W20260618 22:22:05.140625 21337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:22:05.140965 21338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:22:05.141474 21332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:22:05.142550 21332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:22:05.143734 21332 hybrid_clock.cc:648] HybridClock initialized: now 1781821325143708 us; error 39 us; skew 500 ppm
Jun 18 22:22:05 dist-test-slave-g6r4 krb5kdc[21319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821325, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:22:05.146450 21332 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260618 22:22:05.147369 21332 webserver.cc:460] Webserver started at http://127.0.0.1:42443/ using document root <none> and password file <none>
I20260618 22:22:05.147931 21332 fs_manager.cc:359] Metadata directory not provided
I20260618 22:22:05.148029 21332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:22:05.148265 21332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:22:05.150331 21332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781821065380026-393-0/minicluster-data/master-0/data/instance:
uuid: "7c4a4d9b7d404d8aba75167e15554a14"
format_stamp: "Formatted at 2026-06-18 22:22:05 on dist-test-slave-g6r4"
server_key: "435e2d3c8c89db3b9ce8bf0b0e31743e"
server_key_iv: "6036b601fd6a7aa646aa18d957c807ef"
server_key_version: "encryptionkey@0"
I20260618 22:22:05.150893 21332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781821065380026-393-0/minicluster-data/master-0/wal/instance:
uuid: "7c4a4d9b7d404d8aba75167e15554a14"
format_stamp: "Formatted at 2026-06-18 22:22:05 on dist-test-slave-g6r4"
server_key: "435e2d3c8c89db3b9ce8bf0b0e31743e"
server_key_iv: "6036b601fd6a7aa646aa18d957c807ef"
server_key_version: "encryptionkey@0"
I20260618 22:22:05.154500 21332 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:22:05.156797 21347 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:22:05.158010 21332 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:22:05.158159 21332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781821065380026-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781821065380026-393-0/minicluster-data/master-0/wal
uuid: "7c4a4d9b7d404d8aba75167e15554a14"
format_stamp: "Formatted at 2026-06-18 22:22:05 on dist-test-slave-g6r4"
server_key: "435e2d3c8c89db3b9ce8bf0b0e31743e"
server_key_iv: "6036b601fd6a7aa646aa18d957c807ef"
server_key_version: "encryptionkey@0"
I20260618 22:22:05.158268 21332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781821065380026-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781821065380026-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781821065380026-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:22:05.180877 21332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:22:05.183950 21332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:22:05.184147 21332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:22:05.191288 21332 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46477
I20260618 22:22:05.191314 21399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46477 every 8 connection(s)
I20260618 22:22:05.192134 21332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
I20260618 22:22:05.194634 21400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:22:05.199676 21400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c4a4d9b7d404d8aba75167e15554a14: Bootstrap starting.
I20260618 22:22:05.201959 21400 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c4a4d9b7d404d8aba75167e15554a14: Neither blocks nor log segments found. Creating new log.
I20260618 22:22:05.202068 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 21332
I20260618 22:22:05.202216 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781821065380026-393-0/minicluster-data/master-0/wal/instance
I20260618 22:22:05.202499 393 external_mini_cluster.cc:1428] Setting key 69740716a6a3f111b6c29521241b5e14
I20260618 22:22:05.202852 21400 log.cc:824] T 00000000000000000000000000000000 P 7c4a4d9b7d404d8aba75167e15554a14: Log is configured to *not* fsync() on all Append() calls
I20260618 22:22:05.204941 21400 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c4a4d9b7d404d8aba75167e15554a14: No bootstrap required, opened a new log
I20260618 22:22:05.209021 21400 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c4a4d9b7d404d8aba75167e15554a14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c4a4d9b7d404d8aba75167e15554a14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46477 } }
I20260618 22:22:05.209259 21400 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c4a4d9b7d404d8aba75167e15554a14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:22:05.209335 21400 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c4a4d9b7d404d8aba75167e15554a14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c4a4d9b7d404d8aba75167e15554a14, State: Initialized, Role: FOLLOWER
I20260618 22:22:05.209817 21400 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c4a4d9b7d404d8aba75167e15554a14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c4a4d9b7d404d8aba75167e15554a14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46477 } }
I20260618 22:22:05.210016 21400 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c4a4d9b7d404d8aba75167e15554a14 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:22:05.210089 21400 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c4a4d9b7d404d8aba75167e15554a14 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:22:05.210203 21400 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c4a4d9b7d404d8aba75167e15554a14 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:22:05.211022 21400 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c4a4d9b7d404d8aba75167e15554a14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c4a4d9b7d404d8aba75167e15554a14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46477 } }
I20260618 22:22:05.211337 21400 leader_election.cc:302] T 00000000000000000000000000000000 P 7c4a4d9b7d404d8aba75167e15554a14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c4a4d9b7d404d8aba75167e15554a14; no voters:
Jun 18 22:22:05 dist-test-slave-g6r4 krb5kdc[21319](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260618 22:22:05.211618 21400 leader_election.cc:288] T 00000000000000000000000000000000 P 7c4a4d9b7d404d8aba75167e15554a14 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:22:05.211750 21405 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c4a4d9b7d404d8aba75167e15554a14 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:22:05.211985 21405 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c4a4d9b7d404d8aba75167e15554a14 [term 1 LEADER]: Becoming Leader. State: Replica: 7c4a4d9b7d404d8aba75167e15554a14, State: Running, Role: LEADER
I20260618 22:22:05.212357 21405 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c4a4d9b7d404d8aba75167e15554a14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c4a4d9b7d404d8aba75167e15554a14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46477 } }
I20260618 22:22:05.212874 21400 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c4a4d9b7d404d8aba75167e15554a14 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:22:05.213956 21407 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c4a4d9b7d404d8aba75167e15554a14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c4a4d9b7d404d8aba75167e15554a14. Latest consensus state: current_term: 1 leader_uuid: "7c4a4d9b7d404d8aba75167e15554a14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c4a4d9b7d404d8aba75167e15554a14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46477 } } }
I20260618 22:22:05.214123 21407 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c4a4d9b7d404d8aba75167e15554a14 [sys.catalog]: This master's current role is: LEADER
I20260618 22:22:05.214555 21414 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:22:05.214802 21406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c4a4d9b7d404d8aba75167e15554a14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c4a4d9b7d404d8aba75167e15554a14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c4a4d9b7d404d8aba75167e15554a14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46477 } } }
I20260618 22:22:05.215747 21406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c4a4d9b7d404d8aba75167e15554a14 [sys.catalog]: This master's current role is: LEADER
I20260618 22:22:05.217689 21404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:05.203581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.221:48919 (local address 127.0.0.1:46477)
0618 22:22:05.203927 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:05.203944 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:05.203982 (+ 38us) server_negotiation.cc:407] Connection header received
0618 22:22:05.204764 (+ 782us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:05.204786 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:05.205180 (+ 394us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:05.205517 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:05.206365 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:05.207832 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:05.208663 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:05.209345 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:05.209495 (+ 150us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:05.212318 (+ 2823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:22:05.212324 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:22:05.212337 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:22:05.212367 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:22:05.214510 (+ 2143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:05.215086 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:05.215093 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:05.215100 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:05.215232 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:05.215638 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:05.215641 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:05.215643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:05.216077 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:22:05.216309 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0618 22:22:05.216571 (+ 262us) server_negotiation.cc:299] Negotiation successful
0618 22:22:05.216813 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":207,"thread_start_us":87,"threads_started":1}
I20260618 22:22:05.218470 21414 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:22:05.220707 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:22:05.223963 21414 catalog_manager.cc:1347] Generated new cluster ID: f2dcdd6a6f3341e7a1192b4c4db58bcf
I20260618 22:22:05.224016 21414 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:22:05.232645 21414 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:22:05.233073 21404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:05.223225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.221:53123 (local address 127.0.0.1:46477)
0618 22:22:05.223320 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:05.223324 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:05.223532 (+ 208us) server_negotiation.cc:407] Connection header received
0618 22:22:05.223809 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:05.223812 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:05.223857 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:05.223938 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:05.224780 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:05.225615 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:05.226813 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:05.227182 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:05.227255 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:05.229007 (+ 1752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:22:05.229011 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:22:05.229013 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:22:05.229037 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:22:05.230607 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:05.231157 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:05.231161 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:05.231163 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:05.231238 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:05.232110 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:05.232113 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:05.232115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:05.232315 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:22:05.232424 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0618 22:22:05.232839 (+ 415us) server_negotiation.cc:299] Negotiation successful
0618 22:22:05.232929 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260618 22:22:05.233717 21414 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:22:05.243057 21414 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c4a4d9b7d404d8aba75167e15554a14: Generated new TSK 0
I20260618 22:22:05.243839 21414 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:22:05.248587 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 21332
W20260618 22:22:05.253882 21317 connection.cc:535] client connection to 127.0.0.1:46477 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46477): Connection reset by peer (error 104)
2026-06-18T22:22:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1384 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781821065380026-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:22:05 dist-test-slave-g6r4 krb5kdc[21431](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:22:05 dist-test-slave-g6r4 krb5kdc[21431](info): set up 2 sockets
Jun 18 22:22:05 dist-test-slave-g6r4 krb5kdc[21431](info): commencing operation
krb5kdc: starting...
W20260618 22:22:06.274014 393 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781821065380026-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781821065380026-393-0/krb5kdc/test-user.keytab.
Jun 18 22:22:06 dist-test-slave-g6r4 krb5kdc[21431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821326, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:22:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:22:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781821065380026-393-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781821065380026-393-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260618 22:22:06.391160 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781821065380026-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781821065380026-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781821065380026-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45567
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781821065380026-393-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45567
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:22:06.497973 21444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:22:06.498207 21444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:22:06.498245 21444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:22:06.502156 21444 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:22:06.502205 21444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:22:06.502224 21444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:22:06.502240 21444 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:22:06.502255 21444 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:22:06.507249 21444 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781821065380026-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781821065380026-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45567
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781821065380026-393-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781821065380026-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:22:06.508116 21444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:22:06.509092 21444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:22:06.517210 21452 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:22:06.517215 21450 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:22:06.517306 21449 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:22:06.517357 21444 server_base.cc:1034] running on GCE node
I20260618 22:22:06.518008 21444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:22:06.519063 21444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:22:06.520251 21444 hybrid_clock.cc:648] HybridClock initialized: now 1781821326520242 us; error 26 us; skew 500 ppm
Jun 18 22:22:06 dist-test-slave-g6r4 krb5kdc[21431](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821326, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:22:06.525964 21444 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260618 22:22:06.526855 21444 webserver.cc:460] Webserver started at http://127.0.0.1:36585/ using document root <none> and password file <none>
I20260618 22:22:06.527397 21444 fs_manager.cc:359] Metadata directory not provided
I20260618 22:22:06.527486 21444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:22:06.527712 21444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:22:06.529676 21444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781821065380026-393-0/minicluster-data/master-0/data/instance:
uuid: "35db641ba8cf4aea8625aa9edff2e27a"
format_stamp: "Formatted at 2026-06-18 22:22:06 on dist-test-slave-g6r4"
server_key: "f9c9c6df6673ee9295934cc8ca2f35f4"
server_key_iv: "c19e778f9b9c02304668a9b1c3429e08"
server_key_version: "encryptionkey@0"
I20260618 22:22:06.530284 21444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781821065380026-393-0/minicluster-data/master-0/wal/instance:
uuid: "35db641ba8cf4aea8625aa9edff2e27a"
format_stamp: "Formatted at 2026-06-18 22:22:06 on dist-test-slave-g6r4"
server_key: "f9c9c6df6673ee9295934cc8ca2f35f4"
server_key_iv: "c19e778f9b9c02304668a9b1c3429e08"
server_key_version: "encryptionkey@0"
I20260618 22:22:06.533769 21444 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:22:06.536209 21459 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:22:06.537446 21444 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:22:06.537582 21444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781821065380026-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781821065380026-393-0/minicluster-data/master-0/wal
uuid: "35db641ba8cf4aea8625aa9edff2e27a"
format_stamp: "Formatted at 2026-06-18 22:22:06 on dist-test-slave-g6r4"
server_key: "f9c9c6df6673ee9295934cc8ca2f35f4"
server_key_iv: "c19e778f9b9c02304668a9b1c3429e08"
server_key_version: "encryptionkey@0"
I20260618 22:22:06.537695 21444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781821065380026-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781821065380026-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781821065380026-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:22:06.552314 21444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:22:06.555114 21444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:22:06.555316 21444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:22:06.562307 21444 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45567
I20260618 22:22:06.562333 21511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45567 every 8 connection(s)
I20260618 22:22:06.563189 21444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
I20260618 22:22:06.565716 21512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:22:06.567184 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 21444
I20260618 22:22:06.567312 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781821065380026-393-0/minicluster-data/master-0/wal/instance
I20260618 22:22:06.567621 393 external_mini_cluster.cc:1428] Setting key d3e3ecf54c59c4b8bfb966e2e0051fde
I20260618 22:22:06.571581 21512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35db641ba8cf4aea8625aa9edff2e27a: Bootstrap starting.
I20260618 22:22:06.573621 21512 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 35db641ba8cf4aea8625aa9edff2e27a: Neither blocks nor log segments found. Creating new log.
I20260618 22:22:06.574486 21512 log.cc:824] T 00000000000000000000000000000000 P 35db641ba8cf4aea8625aa9edff2e27a: Log is configured to *not* fsync() on all Append() calls
I20260618 22:22:06.576231 21515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:06.568979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.221:55755 (local address 127.0.0.1:45567)
0618 22:22:06.569379 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:06.569393 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:06.569427 (+ 34us) server_negotiation.cc:407] Connection header received
0618 22:22:06.570113 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:06.570131 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:06.570461 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:06.570777 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:06.571153 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:06.572140 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:06.573026 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:06.573490 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:06.573660 (+ 170us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:06.574549 (+ 889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:22:06.574554 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:22:06.574566 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:22:06.575048 (+ 482us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:22:06.575119 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:22:06.575151 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:22:06.575292 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0618 22:22:06.575465 (+ 173us) server_negotiation.cc:299] Negotiation successful
0618 22:22:06.575597 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":112,"threads_started":1}
I20260618 22:22:06.576363 21512 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35db641ba8cf4aea8625aa9edff2e27a: No bootstrap required, opened a new log
I20260618 22:22:06.578997 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:22:06.580155 21512 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35db641ba8cf4aea8625aa9edff2e27a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35db641ba8cf4aea8625aa9edff2e27a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45567 } }
I20260618 22:22:06.580385 21512 raft_consensus.cc:374] T 00000000000000000000000000000000 P 35db641ba8cf4aea8625aa9edff2e27a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:22:06.580463 21512 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35db641ba8cf4aea8625aa9edff2e27a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35db641ba8cf4aea8625aa9edff2e27a, State: Initialized, Role: FOLLOWER
I20260618 22:22:06.580883 21512 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35db641ba8cf4aea8625aa9edff2e27a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35db641ba8cf4aea8625aa9edff2e27a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45567 } }
I20260618 22:22:06.581115 21512 raft_consensus.cc:388] T 00000000000000000000000000000000 P 35db641ba8cf4aea8625aa9edff2e27a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:22:06.581207 21512 raft_consensus.cc:482] T 00000000000000000000000000000000 P 35db641ba8cf4aea8625aa9edff2e27a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:22:06.581312 21512 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35db641ba8cf4aea8625aa9edff2e27a [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:22:06.582233 21512 raft_consensus.cc:504] T 00000000000000000000000000000000 P 35db641ba8cf4aea8625aa9edff2e27a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35db641ba8cf4aea8625aa9edff2e27a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45567 } }
I20260618 22:22:06.582543 21512 leader_election.cc:302] T 00000000000000000000000000000000 P 35db641ba8cf4aea8625aa9edff2e27a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 35db641ba8cf4aea8625aa9edff2e27a; no voters:
I20260618 22:22:06.582847 21512 leader_election.cc:288] T 00000000000000000000000000000000 P 35db641ba8cf4aea8625aa9edff2e27a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:22:06.582916 21522 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 35db641ba8cf4aea8625aa9edff2e27a [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:22:06.583105 21522 raft_consensus.cc:686] T 00000000000000000000000000000000 P 35db641ba8cf4aea8625aa9edff2e27a [term 1 LEADER]: Becoming Leader. State: Replica: 35db641ba8cf4aea8625aa9edff2e27a, State: Running, Role: LEADER
I20260618 22:22:06.583428 21522 consensus_queue.cc:237] T 00000000000000000000000000000000 P 35db641ba8cf4aea8625aa9edff2e27a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35db641ba8cf4aea8625aa9edff2e27a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45567 } }
I20260618 22:22:06.584014 21512 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35db641ba8cf4aea8625aa9edff2e27a [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:22:06.585259 21515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:06.580879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.221:39419 (local address 127.0.0.1:45567)
0618 22:22:06.581000 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:06.581004 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:06.581214 (+ 210us) server_negotiation.cc:407] Connection header received
0618 22:22:06.581329 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:06.581332 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:06.581398 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:06.581479 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:06.581738 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:06.582364 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:06.583426 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:06.583717 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:06.583778 (+ 61us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:06.584641 (+ 863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:22:06.584645 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:22:06.584647 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:22:06.584658 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:22:06.584689 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:22:06.584698 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:22:06.584811 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0618 22:22:06.585046 (+ 235us) server_negotiation.cc:299] Negotiation successful
0618 22:22:06.585086 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260618 22:22:06.585325 21523 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35db641ba8cf4aea8625aa9edff2e27a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "35db641ba8cf4aea8625aa9edff2e27a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35db641ba8cf4aea8625aa9edff2e27a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45567 } } }
I20260618 22:22:06.585440 21523 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35db641ba8cf4aea8625aa9edff2e27a [sys.catalog]: This master's current role is: LEADER
I20260618 22:22:06.585606 21524 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35db641ba8cf4aea8625aa9edff2e27a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35db641ba8cf4aea8625aa9edff2e27a. Latest consensus state: current_term: 1 leader_uuid: "35db641ba8cf4aea8625aa9edff2e27a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "35db641ba8cf4aea8625aa9edff2e27a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45567 } } }
I20260618 22:22:06.586675 21524 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35db641ba8cf4aea8625aa9edff2e27a [sys.catalog]: This master's current role is: LEADER
W20260618 22:22:06.588301 21536 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 35db641ba8cf4aea8625aa9edff2e27a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 22:22:06.588387 21536 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 22:22:06.588464 21537 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:22:06.589339 21537 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:22:06.594772 21537 catalog_manager.cc:1347] Generated new cluster ID: 32c2d3dbfe3d4aa5902a59662becaef3
I20260618 22:22:06.594844 21537 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:22:06.618920 21537 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:22:06.619884 21537 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:22:06.626179 21537 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 35db641ba8cf4aea8625aa9edff2e27a: Generated new TSK 0
I20260618 22:22:06.626819 21537 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:22:06.635234 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 21444
W20260618 22:22:06.640524 21429 connection.cc:535] client connection to 127.0.0.1:45567 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45567): Connection reset by peer (error 104)
2026-06-18T22:22:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1386 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781821065380026-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:22:06 dist-test-slave-g6r4 krb5kdc[21543](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:22:06 dist-test-slave-g6r4 krb5kdc[21543](info): set up 2 sockets
Jun 18 22:22:06 dist-test-slave-g6r4 krb5kdc[21543](info): commencing operation
krb5kdc: starting...
W20260618 22:22:07.666131 393 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781821065380026-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781821065380026-393-0/krb5kdc/test-user.keytab.
Jun 18 22:22:07 dist-test-slave-g6r4 krb5kdc[21543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:22:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:22:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781821065380026-393-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781821065380026-393-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260618 22:22:07.785640 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781821065380026-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781821065380026-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781821065380026-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34593
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781821065380026-393-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42943
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34593
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:22:07.892858 21556 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:22:07.893155 21556 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:22:07.893236 21556 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:22:07.897161 21556 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:22:07.897259 21556 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:22:07.897303 21556 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:22:07.897367 21556 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:22:07.897408 21556 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:22:07.902766 21556 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42943
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781821065380026-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781821065380026-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34593
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781821065380026-393-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21556
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781821065380026-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:22:07.903789 21556 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:22:07.904801 21556 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:22:07.913014 21562 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:22:07.913012 21564 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:22:07.913046 21561 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:22:07.913210 21556 server_base.cc:1034] running on GCE node
I20260618 22:22:07.913858 21556 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:22:07.914911 21556 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:22:07.916066 21556 hybrid_clock.cc:648] HybridClock initialized: now 1781821327916048 us; error 28 us; skew 500 ppm
Jun 18 22:22:07 dist-test-slave-g6r4 krb5kdc[21543](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821327, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:22:07.918684 21556 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260618 22:22:07.919569 21556 webserver.cc:460] Webserver started at http://127.0.0.1:37323/ using document root <none> and password file <none>
I20260618 22:22:07.920090 21556 fs_manager.cc:359] Metadata directory not provided
I20260618 22:22:07.920177 21556 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:22:07.920409 21556 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:22:07.922430 21556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781821065380026-393-0/minicluster-data/master-0/data/instance:
uuid: "99059fb87f3e431e8aaf1276546ae173"
format_stamp: "Formatted at 2026-06-18 22:22:07 on dist-test-slave-g6r4"
server_key: "d6dcc0db9402c3da3ee5f065de535fca"
server_key_iv: "86e0cb49ac3cbd3f622d456391b7d03f"
server_key_version: "encryptionkey@0"
I20260618 22:22:07.923022 21556 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781821065380026-393-0/minicluster-data/master-0/wal/instance:
uuid: "99059fb87f3e431e8aaf1276546ae173"
format_stamp: "Formatted at 2026-06-18 22:22:07 on dist-test-slave-g6r4"
server_key: "d6dcc0db9402c3da3ee5f065de535fca"
server_key_iv: "86e0cb49ac3cbd3f622d456391b7d03f"
server_key_version: "encryptionkey@0"
I20260618 22:22:07.926662 21556 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:22:07.928959 21571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:22:07.930374 21556 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:22:07.930532 21556 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781821065380026-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781821065380026-393-0/minicluster-data/master-0/wal
uuid: "99059fb87f3e431e8aaf1276546ae173"
format_stamp: "Formatted at 2026-06-18 22:22:07 on dist-test-slave-g6r4"
server_key: "d6dcc0db9402c3da3ee5f065de535fca"
server_key_iv: "86e0cb49ac3cbd3f622d456391b7d03f"
server_key_version: "encryptionkey@0"
I20260618 22:22:07.930652 21556 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781821065380026-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781821065380026-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781821065380026-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:22:07.935144 21556 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:22:07.938098 21556 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:22:07.938288 21556 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:22:07.945137 21556 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34593
I20260618 22:22:07.945163 21623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34593 every 8 connection(s)
I20260618 22:22:07.946079 21556 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
I20260618 22:22:07.948649 21624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:22:07.951265 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 21556
I20260618 22:22:07.951417 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781821065380026-393-0/minicluster-data/master-0/wal/instance
I20260618 22:22:07.951690 393 external_mini_cluster.cc:1428] Setting key fcf6eaf1be28e9f014cfda4ff47975e0
I20260618 22:22:07.954371 21624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99059fb87f3e431e8aaf1276546ae173: Bootstrap starting.
I20260618 22:22:07.956945 21624 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 99059fb87f3e431e8aaf1276546ae173: Neither blocks nor log segments found. Creating new log.
I20260618 22:22:07.957011 21627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:07.953035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.221:48433 (local address 127.0.0.1:34593)
0618 22:22:07.953441 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:07.953460 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:07.953500 (+ 40us) server_negotiation.cc:407] Connection header received
0618 22:22:07.954323 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:07.954344 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:07.954694 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:07.955031 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:07.955729 (+ 698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:22:07.955735 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:22:07.955749 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:22:07.955760 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:22:07.955819 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:22:07.955844 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:22:07.955970 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0618 22:22:07.956030 (+ 60us) server_negotiation.cc:299] Negotiation successful
0618 22:22:07.956169 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":122,"threads_started":1}
I20260618 22:22:07.957680 21624 log.cc:824] T 00000000000000000000000000000000 P 99059fb87f3e431e8aaf1276546ae173: Log is configured to *not* fsync() on all Append() calls
I20260618 22:22:07.959619 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:22:07.959758 21624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 99059fb87f3e431e8aaf1276546ae173: No bootstrap required, opened a new log
I20260618 22:22:07.963193 21627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:07.961587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.221:47399 (local address 127.0.0.1:34593)
0618 22:22:07.961708 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:07.961712 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:07.961891 (+ 179us) server_negotiation.cc:407] Connection header received
0618 22:22:07.962027 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:07.962029 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:07.962055 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:07.962118 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:07.962799 (+ 681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:22:07.962802 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:22:07.962804 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:22:07.962814 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:22:07.962835 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:22:07.962845 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:22:07.962913 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0618 22:22:07.963015 (+ 102us) server_negotiation.cc:299] Negotiation successful
0618 22:22:07.963046 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260618 22:22:07.963855 21624 raft_consensus.cc:348] T 00000000000000000000000000000000 P 99059fb87f3e431e8aaf1276546ae173 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99059fb87f3e431e8aaf1276546ae173" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34593 } }
I20260618 22:22:07.964108 21624 raft_consensus.cc:374] T 00000000000000000000000000000000 P 99059fb87f3e431e8aaf1276546ae173 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:22:07.964181 21624 raft_consensus.cc:729] T 00000000000000000000000000000000 P 99059fb87f3e431e8aaf1276546ae173 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99059fb87f3e431e8aaf1276546ae173, State: Initialized, Role: FOLLOWER
I20260618 22:22:07.964565 21624 consensus_queue.cc:260] T 00000000000000000000000000000000 P 99059fb87f3e431e8aaf1276546ae173 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99059fb87f3e431e8aaf1276546ae173" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34593 } }
I20260618 22:22:07.964679 21624 raft_consensus.cc:388] T 00000000000000000000000000000000 P 99059fb87f3e431e8aaf1276546ae173 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:22:07.964725 21624 raft_consensus.cc:482] T 00000000000000000000000000000000 P 99059fb87f3e431e8aaf1276546ae173 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:22:07.964787 21624 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 99059fb87f3e431e8aaf1276546ae173 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:22:07.965539 21624 raft_consensus.cc:504] T 00000000000000000000000000000000 P 99059fb87f3e431e8aaf1276546ae173 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99059fb87f3e431e8aaf1276546ae173" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34593 } }
I20260618 22:22:07.965806 21624 leader_election.cc:302] T 00000000000000000000000000000000 P 99059fb87f3e431e8aaf1276546ae173 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 99059fb87f3e431e8aaf1276546ae173; no voters:
I20260618 22:22:07.966089 21624 leader_election.cc:288] T 00000000000000000000000000000000 P 99059fb87f3e431e8aaf1276546ae173 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:22:07.966223 21634 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 99059fb87f3e431e8aaf1276546ae173 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:22:07.966464 21634 raft_consensus.cc:686] T 00000000000000000000000000000000 P 99059fb87f3e431e8aaf1276546ae173 [term 1 LEADER]: Becoming Leader. State: Replica: 99059fb87f3e431e8aaf1276546ae173, State: Running, Role: LEADER
I20260618 22:22:07.966758 21634 consensus_queue.cc:237] T 00000000000000000000000000000000 P 99059fb87f3e431e8aaf1276546ae173 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99059fb87f3e431e8aaf1276546ae173" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34593 } }
I20260618 22:22:07.967211 21624 sys_catalog.cc:564] T 00000000000000000000000000000000 P 99059fb87f3e431e8aaf1276546ae173 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:22:07.968437 21635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99059fb87f3e431e8aaf1276546ae173 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "99059fb87f3e431e8aaf1276546ae173" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99059fb87f3e431e8aaf1276546ae173" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34593 } } }
I20260618 22:22:07.968394 21636 sys_catalog.cc:455] T 00000000000000000000000000000000 P 99059fb87f3e431e8aaf1276546ae173 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 99059fb87f3e431e8aaf1276546ae173. Latest consensus state: current_term: 1 leader_uuid: "99059fb87f3e431e8aaf1276546ae173" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "99059fb87f3e431e8aaf1276546ae173" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34593 } } }
I20260618 22:22:07.968528 21635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99059fb87f3e431e8aaf1276546ae173 [sys.catalog]: This master's current role is: LEADER
I20260618 22:22:07.968528 21636 sys_catalog.cc:458] T 00000000000000000000000000000000 P 99059fb87f3e431e8aaf1276546ae173 [sys.catalog]: This master's current role is: LEADER
I20260618 22:22:07.968986 21644 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:22:07.971889 21644 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:22:07.976997 21644 catalog_manager.cc:1347] Generated new cluster ID: 2c9b88243e7947ab9d6635e0a36562d9
I20260618 22:22:07.977170 21644 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:22:07.995927 21644 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:22:07.996254 21644 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:22:08.005390 21644 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 99059fb87f3e431e8aaf1276546ae173: Generated new TSK 0
I20260618 22:22:08.006101 21644 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:22:08.014796 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 21556
2026-06-18T22:22:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1379 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781821065380026-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:22:08 dist-test-slave-g6r4 krb5kdc[21655](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:22:08 dist-test-slave-g6r4 krb5kdc[21655](info): set up 2 sockets
Jun 18 22:22:08 dist-test-slave-g6r4 krb5kdc[21655](info): commencing operation
krb5kdc: starting...
W20260618 22:22:09.058293 393 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781821065380026-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781821065380026-393-0/krb5kdc/test-user.keytab.
Jun 18 22:22:09 dist-test-slave-g6r4 krb5kdc[21655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:22:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:22:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781821065380026-393-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781821065380026-393-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260618 22:22:09.176569 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781821065380026-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781821065380026-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781821065380026-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43999
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781821065380026-393-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42199
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43999
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:22:09.282629 21668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:22:09.282877 21668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:22:09.282919 21668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:22:09.286747 21668 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:22:09.286798 21668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:22:09.286816 21668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:22:09.286832 21668 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:22:09.286847 21668 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:22:09.291927 21668 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42199
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781821065380026-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781821065380026-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43999
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781821065380026-393-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21668
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781821065380026-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:22:09.292855 21668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:22:09.293771 21668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:22:09.301790 21676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:22:09.301791 21674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:22:09.301833 21668 server_base.cc:1034] running on GCE node
W20260618 22:22:09.301810 21673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:22:09.302553 21668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:22:09.303593 21668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:22:09.304775 21668 hybrid_clock.cc:648] HybridClock initialized: now 1781821329304763 us; error 24 us; skew 500 ppm
Jun 18 22:22:09 dist-test-slave-g6r4 krb5kdc[21655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821329, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:22:09.307456 21668 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260618 22:22:09.308236 21668 webserver.cc:460] Webserver started at http://127.0.0.1:41777/ using document root <none> and password file <none>
I20260618 22:22:09.308750 21668 fs_manager.cc:359] Metadata directory not provided
I20260618 22:22:09.308837 21668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:22:09.309062 21668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:22:09.310992 21668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781821065380026-393-0/minicluster-data/master-0/data/instance:
uuid: "2464d645278847f39d715f9a728dd509"
format_stamp: "Formatted at 2026-06-18 22:22:09 on dist-test-slave-g6r4"
server_key: "2d86757f0ee8868a64096944a0e9db2c"
server_key_iv: "faed33fcef690ba69c9bed8967d96afa"
server_key_version: "encryptionkey@0"
I20260618 22:22:09.311578 21668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781821065380026-393-0/minicluster-data/master-0/wal/instance:
uuid: "2464d645278847f39d715f9a728dd509"
format_stamp: "Formatted at 2026-06-18 22:22:09 on dist-test-slave-g6r4"
server_key: "2d86757f0ee8868a64096944a0e9db2c"
server_key_iv: "faed33fcef690ba69c9bed8967d96afa"
server_key_version: "encryptionkey@0"
I20260618 22:22:09.314968 21668 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260618 22:22:09.317255 21683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:22:09.318445 21668 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260618 22:22:09.318574 21668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781821065380026-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781821065380026-393-0/minicluster-data/master-0/wal
uuid: "2464d645278847f39d715f9a728dd509"
format_stamp: "Formatted at 2026-06-18 22:22:09 on dist-test-slave-g6r4"
server_key: "2d86757f0ee8868a64096944a0e9db2c"
server_key_iv: "faed33fcef690ba69c9bed8967d96afa"
server_key_version: "encryptionkey@0"
I20260618 22:22:09.318678 21668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781821065380026-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781821065380026-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781821065380026-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:22:09.331552 21668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:22:09.334391 21668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:22:09.334693 21668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:22:09.341675 21668 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43999
I20260618 22:22:09.341687 21735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43999 every 8 connection(s)
I20260618 22:22:09.342655 21668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
I20260618 22:22:09.345229 21736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:22:09.350428 21736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2464d645278847f39d715f9a728dd509: Bootstrap starting.
I20260618 22:22:09.352285 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 21668
I20260618 22:22:09.352414 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781821065380026-393-0/minicluster-data/master-0/wal/instance
I20260618 22:22:09.352717 393 external_mini_cluster.cc:1428] Setting key 07ac5f5524c2aca04e23436e8ac3f106
I20260618 22:22:09.353051 21736 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2464d645278847f39d715f9a728dd509: Neither blocks nor log segments found. Creating new log.
I20260618 22:22:09.353971 21736 log.cc:824] T 00000000000000000000000000000000 P 2464d645278847f39d715f9a728dd509: Log is configured to *not* fsync() on all Append() calls
I20260618 22:22:09.355960 21736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2464d645278847f39d715f9a728dd509: No bootstrap required, opened a new log
I20260618 22:22:09.360337 21736 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2464d645278847f39d715f9a728dd509 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2464d645278847f39d715f9a728dd509" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43999 } }
I20260618 22:22:09.360646 21736 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2464d645278847f39d715f9a728dd509 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:22:09.360702 21736 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2464d645278847f39d715f9a728dd509 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2464d645278847f39d715f9a728dd509, State: Initialized, Role: FOLLOWER
I20260618 22:22:09.361171 21736 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2464d645278847f39d715f9a728dd509 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2464d645278847f39d715f9a728dd509" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43999 } }
I20260618 22:22:09.361296 21736 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2464d645278847f39d715f9a728dd509 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:22:09.361364 21736 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2464d645278847f39d715f9a728dd509 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:22:09.361488 21736 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2464d645278847f39d715f9a728dd509 [term 0 FOLLOWER]: Advancing to term 1
Jun 18 22:22:09 dist-test-slave-g6r4 krb5kdc[21655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260618 22:22:09.362494 21736 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2464d645278847f39d715f9a728dd509 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2464d645278847f39d715f9a728dd509" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43999 } }
I20260618 22:22:09.362855 21736 leader_election.cc:302] T 00000000000000000000000000000000 P 2464d645278847f39d715f9a728dd509 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2464d645278847f39d715f9a728dd509; no voters:
I20260618 22:22:09.363199 21736 leader_election.cc:288] T 00000000000000000000000000000000 P 2464d645278847f39d715f9a728dd509 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:22:09.363265 21741 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2464d645278847f39d715f9a728dd509 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:22:09.363467 21741 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2464d645278847f39d715f9a728dd509 [term 1 LEADER]: Becoming Leader. State: Replica: 2464d645278847f39d715f9a728dd509, State: Running, Role: LEADER
I20260618 22:22:09.363744 21741 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2464d645278847f39d715f9a728dd509 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2464d645278847f39d715f9a728dd509" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43999 } }
I20260618 22:22:09.364423 21736 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2464d645278847f39d715f9a728dd509 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:22:09.365249 21742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2464d645278847f39d715f9a728dd509 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2464d645278847f39d715f9a728dd509. Latest consensus state: current_term: 1 leader_uuid: "2464d645278847f39d715f9a728dd509" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2464d645278847f39d715f9a728dd509" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43999 } } }
I20260618 22:22:09.365388 21742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2464d645278847f39d715f9a728dd509 [sys.catalog]: This master's current role is: LEADER
I20260618 22:22:09.365707 21743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2464d645278847f39d715f9a728dd509 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2464d645278847f39d715f9a728dd509" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2464d645278847f39d715f9a728dd509" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43999 } } }
I20260618 22:22:09.365804 21743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2464d645278847f39d715f9a728dd509 [sys.catalog]: This master's current role is: LEADER
I20260618 22:22:09.366776 21750 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:22:09.368819 21739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:09.353810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.221:52781 (local address 127.0.0.1:43999)
0618 22:22:09.354249 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:09.354263 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:09.354296 (+ 33us) server_negotiation.cc:407] Connection header received
0618 22:22:09.355053 (+ 757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:09.355071 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:09.355384 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:09.355647 (+ 263us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:09.356548 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:09.357491 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:09.358721 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:09.359170 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:09.359283 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:09.362448 (+ 3165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:22:09.362454 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:22:09.362469 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:22:09.362500 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:22:09.365246 (+ 2746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:09.365987 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:09.365993 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:09.365999 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:09.366147 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:09.366622 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:09.366626 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:09.366628 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:09.367165 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:22:09.367409 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0618 22:22:09.367891 (+ 482us) server_negotiation.cc:299] Negotiation successful
0618 22:22:09.368130 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":144,"threads_started":1}
I20260618 22:22:09.369694 21750 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:22:09.371762 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:22:09.375442 21750 catalog_manager.cc:1347] Generated new cluster ID: 9bc9dbd372c2447c90e678ef5df3de09
I20260618 22:22:09.375505 21750 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:22:09.382531 21739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:09.374312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.221:35737 (local address 127.0.0.1:43999)
0618 22:22:09.374416 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:09.374421 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:09.374646 (+ 225us) server_negotiation.cc:407] Connection header received
0618 22:22:09.374767 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:09.374770 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:09.374817 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:09.374903 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:09.375843 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:09.376367 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:09.377227 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:09.377472 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:09.377594 (+ 122us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:09.379009 (+ 1415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:22:09.379014 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:22:09.379017 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:22:09.379042 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:22:09.380566 (+ 1524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:09.381106 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:09.381108 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:09.381109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:09.381184 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:09.381548 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:09.381552 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:09.381554 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:09.381737 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:22:09.381902 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0618 22:22:09.382231 (+ 329us) server_negotiation.cc:299] Negotiation successful
0618 22:22:09.382325 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260618 22:22:09.405843 21750 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:22:09.406744 21750 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:22:09.414235 21750 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2464d645278847f39d715f9a728dd509: Generated new TSK 0
I20260618 22:22:09.414940 21750 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:22:09.420760 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 21668
W20260618 22:22:09.425279 21653 connection.cc:535] client connection to 127.0.0.1:43999 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43999): Connection reset by peer (error 104)
2026-06-18T22:22:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1405 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781821065380026-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:22:09 dist-test-slave-g6r4 krb5kdc[21767](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:22:09 dist-test-slave-g6r4 krb5kdc[21767](info): set up 2 sockets
Jun 18 22:22:09 dist-test-slave-g6r4 krb5kdc[21767](info): commencing operation
krb5kdc: starting...
W20260618 22:22:10.470125 393 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781821065380026-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781821065380026-393-0/krb5kdc/test-user.keytab.
Jun 18 22:22:10 dist-test-slave-g6r4 krb5kdc[21767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:22:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:22:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781821065380026-393-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781821065380026-393-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260618 22:22:10.586341 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781821065380026-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781821065380026-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781821065380026-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37823
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781821065380026-393-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44477
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37823
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:22:10.695750 21780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:22:10.696013 21780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:22:10.696056 21780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:22:10.699862 21780 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:22:10.699913 21780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:22:10.699934 21780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:22:10.699954 21780 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:22:10.699971 21780 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:22:10.704986 21780 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44477
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781821065380026-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781821065380026-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37823
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781821065380026-393-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37823
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781821065380026-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:22:10.705941 21780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:22:10.706889 21780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:22:10.715291 21785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:22:10.715306 21788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:22:10.715306 21786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:22:10.715420 21780 server_base.cc:1034] running on GCE node
I20260618 22:22:10.716173 21780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:22:10.717175 21780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:22:10.718328 21780 hybrid_clock.cc:648] HybridClock initialized: now 1781821330718299 us; error 40 us; skew 500 ppm
Jun 18 22:22:10 dist-test-slave-g6r4 krb5kdc[21767](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821330, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:22:10.721046 21780 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260618 22:22:10.721869 21780 webserver.cc:460] Webserver started at http://127.0.0.1:39013/ using document root <none> and password file <none>
I20260618 22:22:10.722424 21780 fs_manager.cc:359] Metadata directory not provided
I20260618 22:22:10.722482 21780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:22:10.722635 21780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:22:10.724490 21780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781821065380026-393-0/minicluster-data/master-0/data/instance:
uuid: "0ec9e4e6004f434b9feb901ad51e1f73"
format_stamp: "Formatted at 2026-06-18 22:22:10 on dist-test-slave-g6r4"
server_key: "69c09286ffc2f982cd2be8a14da5da15"
server_key_iv: "eb42803923fb7961ba56ca5b5010db10"
server_key_version: "encryptionkey@0"
I20260618 22:22:10.724970 21780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781821065380026-393-0/minicluster-data/master-0/wal/instance:
uuid: "0ec9e4e6004f434b9feb901ad51e1f73"
format_stamp: "Formatted at 2026-06-18 22:22:10 on dist-test-slave-g6r4"
server_key: "69c09286ffc2f982cd2be8a14da5da15"
server_key_iv: "eb42803923fb7961ba56ca5b5010db10"
server_key_version: "encryptionkey@0"
I20260618 22:22:10.728477 21780 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260618 22:22:10.730794 21795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:22:10.731958 21780 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:22:10.732091 21780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781821065380026-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781821065380026-393-0/minicluster-data/master-0/wal
uuid: "0ec9e4e6004f434b9feb901ad51e1f73"
format_stamp: "Formatted at 2026-06-18 22:22:10 on dist-test-slave-g6r4"
server_key: "69c09286ffc2f982cd2be8a14da5da15"
server_key_iv: "eb42803923fb7961ba56ca5b5010db10"
server_key_version: "encryptionkey@0"
I20260618 22:22:10.732223 21780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781821065380026-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781821065380026-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781821065380026-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:22:10.757571 21780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:22:10.760277 21780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:22:10.760469 21780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:22:10.767338 21780 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37823
I20260618 22:22:10.767367 21847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37823 every 8 connection(s)
I20260618 22:22:10.768231 21780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
I20260618 22:22:10.770851 21848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:22:10.772625 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 21780
I20260618 22:22:10.772764 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781821065380026-393-0/minicluster-data/master-0/wal/instance
I20260618 22:22:10.773036 393 external_mini_cluster.cc:1428] Setting key 43eab8acd5e8d3a8e701c28b678ff03f
I20260618 22:22:10.777554 21848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ec9e4e6004f434b9feb901ad51e1f73: Bootstrap starting.
I20260618 22:22:10.780411 21848 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ec9e4e6004f434b9feb901ad51e1f73: Neither blocks nor log segments found. Creating new log.
I20260618 22:22:10.781788 21848 log.cc:824] T 00000000000000000000000000000000 P 0ec9e4e6004f434b9feb901ad51e1f73: Log is configured to *not* fsync() on all Append() calls
I20260618 22:22:10.783939 21851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:10.774450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.221:33369 (local address 127.0.0.1:37823)
0618 22:22:10.774854 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:10.774872 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:10.774912 (+ 40us) server_negotiation.cc:407] Connection header received
0618 22:22:10.775708 (+ 796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:10.775734 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:10.776154 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:10.776510 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:10.776916 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:10.778328 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:10.779606 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:10.780299 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:10.780488 (+ 189us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:10.782008 (+ 1520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:22:10.782015 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:22:10.782031 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:22:10.782497 (+ 466us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:22:10.782586 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:22:10.782619 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:22:10.782765 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0618 22:22:10.782917 (+ 152us) server_negotiation.cc:299] Negotiation successful
0618 22:22:10.783040 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":268,"thread_start_us":103,"threads_started":1}
I20260618 22:22:10.784075 21848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ec9e4e6004f434b9feb901ad51e1f73: No bootstrap required, opened a new log
I20260618 22:22:10.786782 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:22:10.788039 21848 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0ec9e4e6004f434b9feb901ad51e1f73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ec9e4e6004f434b9feb901ad51e1f73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37823 } }
I20260618 22:22:10.788254 21848 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0ec9e4e6004f434b9feb901ad51e1f73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:22:10.788327 21848 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0ec9e4e6004f434b9feb901ad51e1f73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ec9e4e6004f434b9feb901ad51e1f73, State: Initialized, Role: FOLLOWER
I20260618 22:22:10.788803 21848 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ec9e4e6004f434b9feb901ad51e1f73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ec9e4e6004f434b9feb901ad51e1f73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37823 } }
I20260618 22:22:10.788959 21848 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0ec9e4e6004f434b9feb901ad51e1f73 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:22:10.789036 21848 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0ec9e4e6004f434b9feb901ad51e1f73 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:22:10.789137 21848 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0ec9e4e6004f434b9feb901ad51e1f73 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:22:10.790304 21848 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0ec9e4e6004f434b9feb901ad51e1f73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ec9e4e6004f434b9feb901ad51e1f73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37823 } }
I20260618 22:22:10.790654 21848 leader_election.cc:302] T 00000000000000000000000000000000 P 0ec9e4e6004f434b9feb901ad51e1f73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0ec9e4e6004f434b9feb901ad51e1f73; no voters:
I20260618 22:22:10.790992 21848 leader_election.cc:288] T 00000000000000000000000000000000 P 0ec9e4e6004f434b9feb901ad51e1f73 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:22:10.791061 21858 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0ec9e4e6004f434b9feb901ad51e1f73 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:22:10.791265 21858 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0ec9e4e6004f434b9feb901ad51e1f73 [term 1 LEADER]: Becoming Leader. State: Replica: 0ec9e4e6004f434b9feb901ad51e1f73, State: Running, Role: LEADER
I20260618 22:22:10.791603 21858 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ec9e4e6004f434b9feb901ad51e1f73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ec9e4e6004f434b9feb901ad51e1f73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37823 } }
I20260618 22:22:10.792255 21848 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0ec9e4e6004f434b9feb901ad51e1f73 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:22:10.793311 21851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:10.788749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.221:47073 (local address 127.0.0.1:37823)
0618 22:22:10.788863 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:10.788868 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:10.789019 (+ 151us) server_negotiation.cc:407] Connection header received
0618 22:22:10.789159 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:10.789162 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:10.789225 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:10.789313 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:10.789520 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:10.790273 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:10.791384 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:10.791697 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:10.791770 (+ 73us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:10.792709 (+ 939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:22:10.792713 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:22:10.792715 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:22:10.792727 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:22:10.792757 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:22:10.792772 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:22:10.792871 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0618 22:22:10.793124 (+ 253us) server_negotiation.cc:299] Negotiation successful
0618 22:22:10.793167 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260618 22:22:10.794438 21859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ec9e4e6004f434b9feb901ad51e1f73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0ec9e4e6004f434b9feb901ad51e1f73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ec9e4e6004f434b9feb901ad51e1f73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37823 } } }
I20260618 22:22:10.793277 21860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ec9e4e6004f434b9feb901ad51e1f73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ec9e4e6004f434b9feb901ad51e1f73. Latest consensus state: current_term: 1 leader_uuid: "0ec9e4e6004f434b9feb901ad51e1f73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0ec9e4e6004f434b9feb901ad51e1f73" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37823 } } }
I20260618 22:22:10.795039 21860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ec9e4e6004f434b9feb901ad51e1f73 [sys.catalog]: This master's current role is: LEADER
I20260618 22:22:10.794689 21859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ec9e4e6004f434b9feb901ad51e1f73 [sys.catalog]: This master's current role is: LEADER
W20260618 22:22:10.797175 21872 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0ec9e4e6004f434b9feb901ad51e1f73: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 22:22:10.797258 21872 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 22:22:10.797326 21874 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:22:10.798048 21874 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:22:10.802964 21874 catalog_manager.cc:1347] Generated new cluster ID: 33e4cbe96ad3493d95b6c145fce4df1f
I20260618 22:22:10.803048 21874 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:22:10.818619 21874 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:22:10.819559 21874 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:22:10.824756 21874 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0ec9e4e6004f434b9feb901ad51e1f73: Generated new TSK 0
I20260618 22:22:10.825352 21874 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:22:10.837077 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 21780
W20260618 22:22:10.842505 21765 connection.cc:535] client connection to 127.0.0.1:37823 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37823): Connection reset by peer (error 104)
2026-06-18T22:22:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1417 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781821065380026-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:22:10 dist-test-slave-g6r4 krb5kdc[21879](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:22:10 dist-test-slave-g6r4 krb5kdc[21879](info): set up 2 sockets
Jun 18 22:22:10 dist-test-slave-g6r4 krb5kdc[21879](info): commencing operation
krb5kdc: starting...
W20260618 22:22:11.870175 393 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781821065380026-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781821065380026-393-0/krb5kdc/test-user.keytab.
Jun 18 22:22:11 dist-test-slave-g6r4 krb5kdc[21879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821331, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:22:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:22:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781821065380026-393-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781821065380026-393-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260618 22:22:11.985260 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781821065380026-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781821065380026-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781821065380026-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44407
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781821065380026-393-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44407
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:22:12.089293 21892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:22:12.089566 21892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:22:12.089637 21892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:22:12.093510 21892 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:22:12.093602 21892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:22:12.093640 21892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:22:12.093667 21892 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:22:12.093688 21892 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:22:12.098730 21892 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781821065380026-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781821065380026-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44407
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781821065380026-393-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21892
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781821065380026-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:22:12.099695 21892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:22:12.100661 21892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:22:12.108863 21898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:22:12.109061 21892 server_base.cc:1034] running on GCE node
W20260618 22:22:12.108976 21900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:22:12.108863 21897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:22:12.109723 21892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:22:12.110792 21892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:22:12.111950 21892 hybrid_clock.cc:648] HybridClock initialized: now 1781821332111938 us; error 25 us; skew 500 ppm
Jun 18 22:22:12 dist-test-slave-g6r4 krb5kdc[21879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821332, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:22:12.117091 21892 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260618 22:22:12.117882 21892 webserver.cc:460] Webserver started at http://127.0.0.1:34507/ using document root <none> and password file <none>
I20260618 22:22:12.118453 21892 fs_manager.cc:359] Metadata directory not provided
I20260618 22:22:12.118511 21892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:22:12.118665 21892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:22:12.120626 21892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781821065380026-393-0/minicluster-data/master-0/data/instance:
uuid: "7b838daac00f433ebe4ac3039c2a91f4"
format_stamp: "Formatted at 2026-06-18 22:22:12 on dist-test-slave-g6r4"
server_key: "584ec635b84bae7b9cec1472d6013400"
server_key_iv: "12479348f2aa73bc6ba7abd46a15cd3f"
server_key_version: "encryptionkey@0"
I20260618 22:22:12.121104 21892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781821065380026-393-0/minicluster-data/master-0/wal/instance:
uuid: "7b838daac00f433ebe4ac3039c2a91f4"
format_stamp: "Formatted at 2026-06-18 22:22:12 on dist-test-slave-g6r4"
server_key: "584ec635b84bae7b9cec1472d6013400"
server_key_iv: "12479348f2aa73bc6ba7abd46a15cd3f"
server_key_version: "encryptionkey@0"
I20260618 22:22:12.124608 21892 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260618 22:22:12.126807 21907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:22:12.127964 21892 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260618 22:22:12.128057 21892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781821065380026-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781821065380026-393-0/minicluster-data/master-0/wal
uuid: "7b838daac00f433ebe4ac3039c2a91f4"
format_stamp: "Formatted at 2026-06-18 22:22:12 on dist-test-slave-g6r4"
server_key: "584ec635b84bae7b9cec1472d6013400"
server_key_iv: "12479348f2aa73bc6ba7abd46a15cd3f"
server_key_version: "encryptionkey@0"
I20260618 22:22:12.128153 21892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781821065380026-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781821065380026-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781821065380026-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:22:12.132403 21892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:22:12.135008 21892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:22:12.135149 21892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:22:12.141544 21892 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44407
I20260618 22:22:12.141628 21959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44407 every 8 connection(s)
I20260618 22:22:12.142421 21892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
I20260618 22:22:12.144932 21960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:22:12.150122 21960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b838daac00f433ebe4ac3039c2a91f4: Bootstrap starting.
I20260618 22:22:12.150202 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 21892
I20260618 22:22:12.150360 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781821065380026-393-0/minicluster-data/master-0/wal/instance
I20260618 22:22:12.150624 393 external_mini_cluster.cc:1428] Setting key 7264ec1f92618451b6c63e58fc2b1e2a
I20260618 22:22:12.152526 21960 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b838daac00f433ebe4ac3039c2a91f4: Neither blocks nor log segments found. Creating new log.
I20260618 22:22:12.153227 21960 log.cc:824] T 00000000000000000000000000000000 P 7b838daac00f433ebe4ac3039c2a91f4: Log is configured to *not* fsync() on all Append() calls
I20260618 22:22:12.155392 21960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b838daac00f433ebe4ac3039c2a91f4: No bootstrap required, opened a new log
I20260618 22:22:12.155475 21963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:12.151800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.221:56557 (local address 127.0.0.1:44407)
0618 22:22:12.152176 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:12.152189 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:12.152220 (+ 31us) server_negotiation.cc:407] Connection header received
0618 22:22:12.152842 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:12.152859 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:12.153181 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:12.153485 (+ 304us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:12.154236 (+ 751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:22:12.154242 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:22:12.154255 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:22:12.154267 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:22:12.154323 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:22:12.154347 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:22:12.154453 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0618 22:22:12.154542 (+ 89us) server_negotiation.cc:299] Negotiation successful
0618 22:22:12.154683 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":114,"threads_started":1}
I20260618 22:22:12.158049 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:22:12.159231 21960 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b838daac00f433ebe4ac3039c2a91f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b838daac00f433ebe4ac3039c2a91f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44407 } }
I20260618 22:22:12.159451 21960 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b838daac00f433ebe4ac3039c2a91f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:22:12.159524 21960 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b838daac00f433ebe4ac3039c2a91f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b838daac00f433ebe4ac3039c2a91f4, State: Initialized, Role: FOLLOWER
I20260618 22:22:12.159945 21960 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b838daac00f433ebe4ac3039c2a91f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b838daac00f433ebe4ac3039c2a91f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44407 } }
I20260618 22:22:12.160095 21960 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b838daac00f433ebe4ac3039c2a91f4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:22:12.160168 21960 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b838daac00f433ebe4ac3039c2a91f4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:22:12.160269 21960 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b838daac00f433ebe4ac3039c2a91f4 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:22:12.161440 21960 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b838daac00f433ebe4ac3039c2a91f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b838daac00f433ebe4ac3039c2a91f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44407 } }
I20260618 22:22:12.161610 21963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:12.160026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.221:38361 (local address 127.0.0.1:44407)
0618 22:22:12.160142 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:12.160147 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:12.160321 (+ 174us) server_negotiation.cc:407] Connection header received
0618 22:22:12.160452 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:12.160454 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:12.160480 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:12.160542 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:12.161170 (+ 628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:22:12.161173 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:22:12.161175 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:22:12.161184 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:22:12.161206 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:22:12.161216 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:22:12.161282 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0618 22:22:12.161402 (+ 120us) server_negotiation.cc:299] Negotiation successful
0618 22:22:12.161430 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260618 22:22:12.161821 21960 leader_election.cc:302] T 00000000000000000000000000000000 P 7b838daac00f433ebe4ac3039c2a91f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b838daac00f433ebe4ac3039c2a91f4; no voters:
I20260618 22:22:12.162143 21960 leader_election.cc:288] T 00000000000000000000000000000000 P 7b838daac00f433ebe4ac3039c2a91f4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:22:12.162210 21970 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b838daac00f433ebe4ac3039c2a91f4 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:22:12.162410 21970 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b838daac00f433ebe4ac3039c2a91f4 [term 1 LEADER]: Becoming Leader. State: Replica: 7b838daac00f433ebe4ac3039c2a91f4, State: Running, Role: LEADER
I20260618 22:22:12.162722 21970 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b838daac00f433ebe4ac3039c2a91f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b838daac00f433ebe4ac3039c2a91f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44407 } }
I20260618 22:22:12.163234 21960 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b838daac00f433ebe4ac3039c2a91f4 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:22:12.164163 21971 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b838daac00f433ebe4ac3039c2a91f4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b838daac00f433ebe4ac3039c2a91f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b838daac00f433ebe4ac3039c2a91f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44407 } } }
I20260618 22:22:12.164319 21971 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b838daac00f433ebe4ac3039c2a91f4 [sys.catalog]: This master's current role is: LEADER
I20260618 22:22:12.164582 21971 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b838daac00f433ebe4ac3039c2a91f4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b838daac00f433ebe4ac3039c2a91f4. Latest consensus state: current_term: 1 leader_uuid: "7b838daac00f433ebe4ac3039c2a91f4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b838daac00f433ebe4ac3039c2a91f4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44407 } } }
I20260618 22:22:12.164714 21971 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b838daac00f433ebe4ac3039c2a91f4 [sys.catalog]: This master's current role is: LEADER
I20260618 22:22:12.164758 21979 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:22:12.167688 21979 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:22:12.172561 21979 catalog_manager.cc:1347] Generated new cluster ID: 07a94bea72c54736a6fc985e9d9d73a2
I20260618 22:22:12.172624 21979 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:22:12.189083 21979 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:22:12.189380 21979 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:22:12.196871 21979 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b838daac00f433ebe4ac3039c2a91f4: Generated new TSK 0
I20260618 22:22:12.197579 21979 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:22:12.208750 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 21892
2026-06-18T22:22:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1371 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781821065380026-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:22:12 dist-test-slave-g6r4 krb5kdc[21991](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:22:12 dist-test-slave-g6r4 krb5kdc[21991](info): set up 2 sockets
Jun 18 22:22:12 dist-test-slave-g6r4 krb5kdc[21991](info): commencing operation
krb5kdc: starting...
W20260618 22:22:13.246099 393 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781821065380026-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781821065380026-393-0/krb5kdc/test-user.keytab.
Jun 18 22:22:13 dist-test-slave-g6r4 krb5kdc[21991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:22:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:22:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260618 22:22:13.361236 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781821065380026-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781821065380026-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781821065380026-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:44369
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:41573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:44369
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:22:13.466616 22004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:22:13.466854 22004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:22:13.466893 22004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:22:13.470882 22004 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:22:13.470932 22004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:22:13.470952 22004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:22:13.470968 22004 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:22:13.470983 22004 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:22:13.476016 22004 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:41573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781821065380026-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781821065380026-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:44369
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:44369
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781821065380026-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:22:13.477018 22004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:22:13.478078 22004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:22:13.486068 22012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:22:13.486110 22010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:22:13.486068 22009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:22:13.486431 22004 server_base.cc:1034] running on GCE node
I20260618 22:22:13.487028 22004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:22:13.488045 22004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:22:13.489233 22004 hybrid_clock.cc:648] HybridClock initialized: now 1781821333489198 us; error 47 us; skew 500 ppm
Jun 18 22:22:13 dist-test-slave-g6r4 krb5kdc[21991](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821333, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:22:13.491990 22004 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260618 22:22:13.492854 22004 webserver.cc:460] Webserver started at http://127.0.98.126:38557/ using document root <none> and password file <none>
I20260618 22:22:13.493381 22004 fs_manager.cc:359] Metadata directory not provided
I20260618 22:22:13.493469 22004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:22:13.493696 22004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:22:13.495656 22004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781821065380026-393-0/minicluster-data/master-0/data/instance:
uuid: "0f260d60a183401e99d5a38840b08607"
format_stamp: "Formatted at 2026-06-18 22:22:13 on dist-test-slave-g6r4"
server_key: "345ea59c2066e5555440a498df262851"
server_key_iv: "294c1960d4d272c6162bfce3e5b8e349"
server_key_version: "encryptionkey@0"
I20260618 22:22:13.496196 22004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781821065380026-393-0/minicluster-data/master-0/wal/instance:
uuid: "0f260d60a183401e99d5a38840b08607"
format_stamp: "Formatted at 2026-06-18 22:22:13 on dist-test-slave-g6r4"
server_key: "345ea59c2066e5555440a498df262851"
server_key_iv: "294c1960d4d272c6162bfce3e5b8e349"
server_key_version: "encryptionkey@0"
I20260618 22:22:13.499646 22004 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260618 22:22:13.501785 22019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:22:13.502950 22004 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:22:13.503083 22004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781821065380026-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781821065380026-393-0/minicluster-data/master-0/wal
uuid: "0f260d60a183401e99d5a38840b08607"
format_stamp: "Formatted at 2026-06-18 22:22:13 on dist-test-slave-g6r4"
server_key: "345ea59c2066e5555440a498df262851"
server_key_iv: "294c1960d4d272c6162bfce3e5b8e349"
server_key_version: "encryptionkey@0"
I20260618 22:22:13.503190 22004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781821065380026-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781821065380026-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781821065380026-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:22:13.526368 22004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:22:13.529307 22004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:22:13.529498 22004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:22:13.536298 22004 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:44369
I20260618 22:22:13.536319 22071 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:44369 every 8 connection(s)
I20260618 22:22:13.537185 22004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
I20260618 22:22:13.539743 22072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:22:13.544752 22072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f260d60a183401e99d5a38840b08607: Bootstrap starting.
I20260618 22:22:13.546774 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 22004
I20260618 22:22:13.546916 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781821065380026-393-0/minicluster-data/master-0/wal/instance
I20260618 22:22:13.547111 22072 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f260d60a183401e99d5a38840b08607: Neither blocks nor log segments found. Creating new log.
I20260618 22:22:13.547225 393 external_mini_cluster.cc:1428] Setting key 1e748fb60a4ccf7f7e6a8eb2f50c027b
I20260618 22:22:13.547935 22072 log.cc:824] T 00000000000000000000000000000000 P 0f260d60a183401e99d5a38840b08607: Log is configured to *not* fsync() on all Append() calls
I20260618 22:22:13.550164 22072 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f260d60a183401e99d5a38840b08607: No bootstrap required, opened a new log
I20260618 22:22:13.554278 22072 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f260d60a183401e99d5a38840b08607 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f260d60a183401e99d5a38840b08607" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 44369 } }
I20260618 22:22:13.554526 22072 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f260d60a183401e99d5a38840b08607 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:22:13.554868 22072 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f260d60a183401e99d5a38840b08607 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f260d60a183401e99d5a38840b08607, State: Initialized, Role: FOLLOWER
I20260618 22:22:13.555301 22072 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f260d60a183401e99d5a38840b08607 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f260d60a183401e99d5a38840b08607" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 44369 } }
I20260618 22:22:13.555420 22072 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f260d60a183401e99d5a38840b08607 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:22:13.555492 22072 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f260d60a183401e99d5a38840b08607 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:22:13.555606 22072 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f260d60a183401e99d5a38840b08607 [term 0 FOLLOWER]: Advancing to term 1
Jun 18 22:22:13 dist-test-slave-g6r4 krb5kdc[21991](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821333, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:22:13.556677 22072 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f260d60a183401e99d5a38840b08607 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f260d60a183401e99d5a38840b08607" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 44369 } }
I20260618 22:22:13.557029 22072 leader_election.cc:302] T 00000000000000000000000000000000 P 0f260d60a183401e99d5a38840b08607 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f260d60a183401e99d5a38840b08607; no voters:
I20260618 22:22:13.557345 22072 leader_election.cc:288] T 00000000000000000000000000000000 P 0f260d60a183401e99d5a38840b08607 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:22:13.557439 22077 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f260d60a183401e99d5a38840b08607 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:22:13.557663 22077 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f260d60a183401e99d5a38840b08607 [term 1 LEADER]: Becoming Leader. State: Replica: 0f260d60a183401e99d5a38840b08607, State: Running, Role: LEADER
I20260618 22:22:13.557986 22077 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f260d60a183401e99d5a38840b08607 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f260d60a183401e99d5a38840b08607" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 44369 } }
I20260618 22:22:13.558781 22072 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f260d60a183401e99d5a38840b08607 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:22:13.559698 22079 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f260d60a183401e99d5a38840b08607 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f260d60a183401e99d5a38840b08607. Latest consensus state: current_term: 1 leader_uuid: "0f260d60a183401e99d5a38840b08607" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f260d60a183401e99d5a38840b08607" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 44369 } } }
I20260618 22:22:13.559823 22079 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f260d60a183401e99d5a38840b08607 [sys.catalog]: This master's current role is: LEADER
I20260618 22:22:13.560053 22078 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f260d60a183401e99d5a38840b08607 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f260d60a183401e99d5a38840b08607" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f260d60a183401e99d5a38840b08607" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 44369 } } }
I20260618 22:22:13.560175 22078 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f260d60a183401e99d5a38840b08607 [sys.catalog]: This master's current role is: LEADER
I20260618 22:22:13.562738 22076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:13.548289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38587 (local address 127.0.98.126:44369)
0618 22:22:13.548742 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:13.548783 (+ 41us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:13.548823 (+ 40us) server_negotiation.cc:407] Connection header received
0618 22:22:13.549649 (+ 826us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:13.549671 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:13.550082 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:13.550415 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:13.551109 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:13.552573 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:13.553418 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:13.554148 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:13.554311 (+ 163us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:13.557104 (+ 2793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:22:13.557112 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:22:13.557130 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:22:13.557158 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:22:13.559371 (+ 2213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:13.560006 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:13.560012 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:13.560017 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:13.560148 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:13.560561 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:13.560564 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:13.560566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:13.561045 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:22:13.561241 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0618 22:22:13.561615 (+ 374us) server_negotiation.cc:299] Negotiation successful
0618 22:22:13.561825 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":171,"threads_started":1}
W20260618 22:22:13.564186 22092 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0f260d60a183401e99d5a38840b08607: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 22:22:13.564275 22092 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 22:22:13.564368 22086 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:22:13.565241 22086 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:22:13.565864 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:22:13.570938 22086 catalog_manager.cc:1347] Generated new cluster ID: 4d5622c62b30486293654ca0a680cf92
I20260618 22:22:13.570997 22086 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:22:13.576244 22076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:13.567964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37859 (local address 127.0.98.126:44369)
0618 22:22:13.568074 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:13.568078 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:13.568296 (+ 218us) server_negotiation.cc:407] Connection header received
0618 22:22:13.568437 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:13.568441 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:13.568480 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:13.568567 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:13.569320 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:13.570071 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:13.571157 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:13.571421 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:13.571531 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:13.572933 (+ 1402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:22:13.572937 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:22:13.572939 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:22:13.572962 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:22:13.574490 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:13.574953 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:13.574957 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:13.574959 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:13.575042 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:13.575372 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:13.575375 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:13.575377 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:13.575547 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:22:13.575712 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0618 22:22:13.575971 (+ 259us) server_negotiation.cc:299] Negotiation successful
0618 22:22:13.576071 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260618 22:22:13.594609 22086 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:22:13.595536 22086 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:22:13.602347 22086 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f260d60a183401e99d5a38840b08607: Generated new TSK 0
I20260618 22:22:13.602975 22086 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:22:13.612576 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 22004
W20260618 22:22:13.617558 21989 connection.cc:535] client connection to 127.0.98.126:44369 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.126:44369): Connection reset by peer (error 104)
2026-06-18T22:22:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1404 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781821065380026-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:22:13 dist-test-slave-g6r4 krb5kdc[22103](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:22:13 dist-test-slave-g6r4 krb5kdc[22103](info): set up 2 sockets
Jun 18 22:22:13 dist-test-slave-g6r4 krb5kdc[22103](info): commencing operation
krb5kdc: starting...
W20260618 22:22:14.642091 393 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781821065380026-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781821065380026-393-0/krb5kdc/test-user.keytab.
Jun 18 22:22:14 dist-test-slave-g6r4 krb5kdc[22103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821334, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:22:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:22:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260618 22:22:14.758965 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781821065380026-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781821065380026-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781821065380026-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:34721
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:40779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:34721
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:22:14.866650 22116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:22:14.866909 22116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:22:14.866978 22116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:22:14.870939 22116 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:22:14.871014 22116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:22:14.871052 22116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:22:14.871111 22116 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:22:14.871151 22116 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:22:14.876554 22116 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:40779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781821065380026-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781821065380026-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:34721
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:34721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781821065380026-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:22:14.877519 22116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:22:14.878530 22116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:22:14.886495 22121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:22:14.886618 22116 server_base.cc:1034] running on GCE node
W20260618 22:22:14.886554 22124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:22:14.886564 22122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:22:14.887274 22116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:22:14.888317 22116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:22:14.889508 22116 hybrid_clock.cc:648] HybridClock initialized: now 1781821334889478 us; error 42 us; skew 500 ppm
Jun 18 22:22:14 dist-test-slave-g6r4 krb5kdc[22103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821334, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:22:14.892225 22116 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260618 22:22:14.893042 22116 webserver.cc:460] Webserver started at http://127.0.98.126:33387/ using document root <none> and password file <none>
I20260618 22:22:14.893596 22116 fs_manager.cc:359] Metadata directory not provided
I20260618 22:22:14.893687 22116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:22:14.893953 22116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:22:14.895862 22116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781821065380026-393-0/minicluster-data/master-0/data/instance:
uuid: "beeb8e51058d4e8eb5cbe04f8027b848"
format_stamp: "Formatted at 2026-06-18 22:22:14 on dist-test-slave-g6r4"
server_key: "8b88b47fd50f21d56ab85f7359b8f56c"
server_key_iv: "5fa2d53d1190bc330df440e406314418"
server_key_version: "encryptionkey@0"
I20260618 22:22:14.896476 22116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781821065380026-393-0/minicluster-data/master-0/wal/instance:
uuid: "beeb8e51058d4e8eb5cbe04f8027b848"
format_stamp: "Formatted at 2026-06-18 22:22:14 on dist-test-slave-g6r4"
server_key: "8b88b47fd50f21d56ab85f7359b8f56c"
server_key_iv: "5fa2d53d1190bc330df440e406314418"
server_key_version: "encryptionkey@0"
I20260618 22:22:14.901387 22116 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260618 22:22:14.904558 22131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:22:14.905710 22116 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:22:14.905807 22116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781821065380026-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781821065380026-393-0/minicluster-data/master-0/wal
uuid: "beeb8e51058d4e8eb5cbe04f8027b848"
format_stamp: "Formatted at 2026-06-18 22:22:14 on dist-test-slave-g6r4"
server_key: "8b88b47fd50f21d56ab85f7359b8f56c"
server_key_iv: "5fa2d53d1190bc330df440e406314418"
server_key_version: "encryptionkey@0"
I20260618 22:22:14.905886 22116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781821065380026-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781821065380026-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781821065380026-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:22:14.924054 22116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:22:14.926716 22116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:22:14.926877 22116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:22:14.933809 22116 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:34721
I20260618 22:22:14.933884 22183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:34721 every 8 connection(s)
I20260618 22:22:14.934804 22116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
I20260618 22:22:14.937431 22184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:22:14.942615 22184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P beeb8e51058d4e8eb5cbe04f8027b848: Bootstrap starting.
I20260618 22:22:14.944402 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 22116
I20260618 22:22:14.944573 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781821065380026-393-0/minicluster-data/master-0/wal/instance
I20260618 22:22:14.944865 393 external_mini_cluster.cc:1428] Setting key a1a29e55ff250bff409275597392df46
I20260618 22:22:14.945003 22184 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P beeb8e51058d4e8eb5cbe04f8027b848: Neither blocks nor log segments found. Creating new log.
I20260618 22:22:14.945727 22184 log.cc:824] T 00000000000000000000000000000000 P beeb8e51058d4e8eb5cbe04f8027b848: Log is configured to *not* fsync() on all Append() calls
I20260618 22:22:14.947923 22184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P beeb8e51058d4e8eb5cbe04f8027b848: No bootstrap required, opened a new log
I20260618 22:22:14.952140 22184 raft_consensus.cc:348] T 00000000000000000000000000000000 P beeb8e51058d4e8eb5cbe04f8027b848 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beeb8e51058d4e8eb5cbe04f8027b848" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34721 } }
I20260618 22:22:14.952414 22184 raft_consensus.cc:374] T 00000000000000000000000000000000 P beeb8e51058d4e8eb5cbe04f8027b848 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:22:14.952498 22184 raft_consensus.cc:729] T 00000000000000000000000000000000 P beeb8e51058d4e8eb5cbe04f8027b848 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: beeb8e51058d4e8eb5cbe04f8027b848, State: Initialized, Role: FOLLOWER
I20260618 22:22:14.952975 22184 consensus_queue.cc:260] T 00000000000000000000000000000000 P beeb8e51058d4e8eb5cbe04f8027b848 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beeb8e51058d4e8eb5cbe04f8027b848" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34721 } }
I20260618 22:22:14.953125 22184 raft_consensus.cc:388] T 00000000000000000000000000000000 P beeb8e51058d4e8eb5cbe04f8027b848 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:22:14.953190 22184 raft_consensus.cc:482] T 00000000000000000000000000000000 P beeb8e51058d4e8eb5cbe04f8027b848 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:22:14.953284 22184 raft_consensus.cc:3037] T 00000000000000000000000000000000 P beeb8e51058d4e8eb5cbe04f8027b848 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:22:14.954228 22184 raft_consensus.cc:504] T 00000000000000000000000000000000 P beeb8e51058d4e8eb5cbe04f8027b848 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beeb8e51058d4e8eb5cbe04f8027b848" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34721 } }
I20260618 22:22:14.954430 22188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:14.946000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59405 (local address 127.0.98.126:34721)
0618 22:22:14.946376 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:14.946390 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:14.946424 (+ 34us) server_negotiation.cc:407] Connection header received
0618 22:22:14.947248 (+ 824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:14.947271 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:14.947671 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:14.948017 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:14.948436 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:14.949879 (+ 1443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:14.950819 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:14.951537 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:14.951701 (+ 164us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:14.952701 (+ 1000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:22:14.952706 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:22:14.952717 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:22:14.953082 (+ 365us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:22:14.953164 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:22:14.953196 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:22:14.953324 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0618 22:22:14.953488 (+ 164us) server_negotiation.cc:299] Negotiation successful
0618 22:22:14.953622 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":101,"threads_started":1}
I20260618 22:22:14.954596 22184 leader_election.cc:302] T 00000000000000000000000000000000 P beeb8e51058d4e8eb5cbe04f8027b848 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: beeb8e51058d4e8eb5cbe04f8027b848; no voters:
I20260618 22:22:14.954970 22184 leader_election.cc:288] T 00000000000000000000000000000000 P beeb8e51058d4e8eb5cbe04f8027b848 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:22:14.955041 22189 raft_consensus.cc:2781] T 00000000000000000000000000000000 P beeb8e51058d4e8eb5cbe04f8027b848 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:22:14.955245 22189 raft_consensus.cc:686] T 00000000000000000000000000000000 P beeb8e51058d4e8eb5cbe04f8027b848 [term 1 LEADER]: Becoming Leader. State: Replica: beeb8e51058d4e8eb5cbe04f8027b848, State: Running, Role: LEADER
I20260618 22:22:14.956187 22184 sys_catalog.cc:564] T 00000000000000000000000000000000 P beeb8e51058d4e8eb5cbe04f8027b848 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:22:14.956327 22189 consensus_queue.cc:237] T 00000000000000000000000000000000 P beeb8e51058d4e8eb5cbe04f8027b848 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beeb8e51058d4e8eb5cbe04f8027b848" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34721 } }
I20260618 22:22:14.957216 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:22:14.959136 22190 sys_catalog.cc:455] T 00000000000000000000000000000000 P beeb8e51058d4e8eb5cbe04f8027b848 [sys.catalog]: SysCatalogTable state changed. Reason: New leader beeb8e51058d4e8eb5cbe04f8027b848. Latest consensus state: current_term: 1 leader_uuid: "beeb8e51058d4e8eb5cbe04f8027b848" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beeb8e51058d4e8eb5cbe04f8027b848" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34721 } } }
I20260618 22:22:14.959177 22191 sys_catalog.cc:455] T 00000000000000000000000000000000 P beeb8e51058d4e8eb5cbe04f8027b848 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "beeb8e51058d4e8eb5cbe04f8027b848" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "beeb8e51058d4e8eb5cbe04f8027b848" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 34721 } } }
I20260618 22:22:14.959271 22190 sys_catalog.cc:458] T 00000000000000000000000000000000 P beeb8e51058d4e8eb5cbe04f8027b848 [sys.catalog]: This master's current role is: LEADER
I20260618 22:22:14.959293 22191 sys_catalog.cc:458] T 00000000000000000000000000000000 P beeb8e51058d4e8eb5cbe04f8027b848 [sys.catalog]: This master's current role is: LEADER
W20260618 22:22:14.961526 22203 catalog_manager.cc:1558] T 00000000000000000000000000000000 P beeb8e51058d4e8eb5cbe04f8027b848: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 22:22:14.961606 22203 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 22:22:14.961666 22209 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:22:14.962517 22209 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:22:14.963493 22188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:14.959444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49027 (local address 127.0.98.126:34721)
0618 22:22:14.959535 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:14.959540 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:14.959962 (+ 422us) server_negotiation.cc:407] Connection header received
0618 22:22:14.960084 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:14.960087 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:14.960123 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:14.960209 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:14.960461 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:14.960961 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:14.961782 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:14.962122 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:14.962228 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:14.963069 (+ 841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:22:14.963072 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:22:14.963075 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:22:14.963086 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:22:14.963108 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:22:14.963126 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:22:14.963219 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0618 22:22:14.963335 (+ 116us) server_negotiation.cc:299] Negotiation successful
0618 22:22:14.963365 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260618 22:22:14.967418 22209 catalog_manager.cc:1347] Generated new cluster ID: 531da083b9ae45a7a90917fe45759c85
I20260618 22:22:14.967475 22209 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:22:14.976305 22209 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:22:14.977123 22209 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:22:14.981190 22209 catalog_manager.cc:5868] T 00000000000000000000000000000000 P beeb8e51058d4e8eb5cbe04f8027b848: Generated new TSK 0
I20260618 22:22:14.981803 22209 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:22:14.989529 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 22116
W20260618 22:22:14.994338 22101 connection.cc:535] client connection to 127.0.98.126:34721 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.126:34721): Connection reset by peer (error 104)
2026-06-18T22:22:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1376 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781821065380026-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:22:15 dist-test-slave-g6r4 krb5kdc[22215](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:22:15 dist-test-slave-g6r4 krb5kdc[22215](info): set up 2 sockets
Jun 18 22:22:15 dist-test-slave-g6r4 krb5kdc[22215](info): commencing operation
krb5kdc: starting...
W20260618 22:22:16.018170 393 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781821065380026-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781821065380026-393-0/krb5kdc/test-user.keytab.
Jun 18 22:22:16 dist-test-slave-g6r4 krb5kdc[22215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:22:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:22:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260618 22:22:16.133229 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781821065380026-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781821065380026-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781821065380026-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:46797
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:39709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:46797
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:22:16.237874 22228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:22:16.238183 22228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:22:16.238268 22228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:22:16.242398 22228 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:22:16.242477 22228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:22:16.242515 22228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:22:16.242578 22228 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:22:16.242615 22228 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:22:16.247670 22228 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:39709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781821065380026-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781821065380026-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:46797
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:46797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781821065380026-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:22:16.248633 22228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:22:16.249590 22228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:22:16.257613 22236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:22:16.257601 22234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:22:16.257617 22233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:22:16.258069 22228 server_base.cc:1034] running on GCE node
I20260618 22:22:16.258549 22228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:22:16.259519 22228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:22:16.260746 22228 hybrid_clock.cc:648] HybridClock initialized: now 1781821336260718 us; error 42 us; skew 500 ppm
Jun 18 22:22:16 dist-test-slave-g6r4 krb5kdc[22215](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821336, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:22:16.263545 22228 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260618 22:22:16.264389 22228 webserver.cc:460] Webserver started at http://127.0.98.126:36911/ using document root <none> and password file <none>
I20260618 22:22:16.264863 22228 fs_manager.cc:359] Metadata directory not provided
I20260618 22:22:16.264925 22228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:22:16.265075 22228 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:22:16.266981 22228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781821065380026-393-0/minicluster-data/master-0/data/instance:
uuid: "786730d6bce442ab8790b928b90b94b2"
format_stamp: "Formatted at 2026-06-18 22:22:16 on dist-test-slave-g6r4"
server_key: "840d41c5d4130c3dcf71e80e950a1a5f"
server_key_iv: "b8f347ce3a4fd73db8f3e633378fdfc7"
server_key_version: "encryptionkey@0"
I20260618 22:22:16.267462 22228 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781821065380026-393-0/minicluster-data/master-0/wal/instance:
uuid: "786730d6bce442ab8790b928b90b94b2"
format_stamp: "Formatted at 2026-06-18 22:22:16 on dist-test-slave-g6r4"
server_key: "840d41c5d4130c3dcf71e80e950a1a5f"
server_key_iv: "b8f347ce3a4fd73db8f3e633378fdfc7"
server_key_version: "encryptionkey@0"
I20260618 22:22:16.270929 22228 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:22:16.273169 22243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:22:16.274456 22228 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:22:16.274616 22228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781821065380026-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781821065380026-393-0/minicluster-data/master-0/wal
uuid: "786730d6bce442ab8790b928b90b94b2"
format_stamp: "Formatted at 2026-06-18 22:22:16 on dist-test-slave-g6r4"
server_key: "840d41c5d4130c3dcf71e80e950a1a5f"
server_key_iv: "b8f347ce3a4fd73db8f3e633378fdfc7"
server_key_version: "encryptionkey@0"
I20260618 22:22:16.274708 22228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781821065380026-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781821065380026-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781821065380026-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:22:16.279126 22228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:22:16.282449 22228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:22:16.282644 22228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:22:16.289283 22228 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:46797
I20260618 22:22:16.289295 22295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:46797 every 8 connection(s)
I20260618 22:22:16.290184 22228 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
I20260618 22:22:16.292761 22296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:22:16.298183 22296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 786730d6bce442ab8790b928b90b94b2: Bootstrap starting.
I20260618 22:22:16.298215 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 22228
I20260618 22:22:16.298400 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781821065380026-393-0/minicluster-data/master-0/wal/instance
I20260618 22:22:16.298676 393 external_mini_cluster.cc:1428] Setting key ae276beffe392617e55bc224bf203075
I20260618 22:22:16.300628 22296 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 786730d6bce442ab8790b928b90b94b2: Neither blocks nor log segments found. Creating new log.
I20260618 22:22:16.301311 22296 log.cc:824] T 00000000000000000000000000000000 P 786730d6bce442ab8790b928b90b94b2: Log is configured to *not* fsync() on all Append() calls
I20260618 22:22:16.303411 22296 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 786730d6bce442ab8790b928b90b94b2: No bootstrap required, opened a new log
I20260618 22:22:16.303782 22299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:16.299862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37119 (local address 127.0.98.126:46797)
0618 22:22:16.300282 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:16.300296 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:16.300367 (+ 71us) server_negotiation.cc:407] Connection header received
0618 22:22:16.301017 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:16.301035 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:16.301317 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:16.301672 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:16.302478 (+ 806us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:22:16.302483 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:22:16.302497 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:22:16.302508 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:22:16.302567 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:22:16.302592 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:22:16.302703 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0618 22:22:16.302781 (+ 78us) server_negotiation.cc:299] Negotiation successful
0618 22:22:16.302922 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":130,"threads_started":1}
I20260618 22:22:16.306460 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:22:16.307339 22296 raft_consensus.cc:348] T 00000000000000000000000000000000 P 786730d6bce442ab8790b928b90b94b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "786730d6bce442ab8790b928b90b94b2" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46797 } }
I20260618 22:22:16.307600 22296 raft_consensus.cc:374] T 00000000000000000000000000000000 P 786730d6bce442ab8790b928b90b94b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:22:16.307709 22296 raft_consensus.cc:729] T 00000000000000000000000000000000 P 786730d6bce442ab8790b928b90b94b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 786730d6bce442ab8790b928b90b94b2, State: Initialized, Role: FOLLOWER
I20260618 22:22:16.308120 22296 consensus_queue.cc:260] T 00000000000000000000000000000000 P 786730d6bce442ab8790b928b90b94b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "786730d6bce442ab8790b928b90b94b2" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46797 } }
I20260618 22:22:16.308288 22296 raft_consensus.cc:388] T 00000000000000000000000000000000 P 786730d6bce442ab8790b928b90b94b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:22:16.308367 22296 raft_consensus.cc:482] T 00000000000000000000000000000000 P 786730d6bce442ab8790b928b90b94b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:22:16.308507 22296 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 786730d6bce442ab8790b928b90b94b2 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:22:16.309378 22296 raft_consensus.cc:504] T 00000000000000000000000000000000 P 786730d6bce442ab8790b928b90b94b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "786730d6bce442ab8790b928b90b94b2" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46797 } }
I20260618 22:22:16.309692 22296 leader_election.cc:302] T 00000000000000000000000000000000 P 786730d6bce442ab8790b928b90b94b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 786730d6bce442ab8790b928b90b94b2; no voters:
I20260618 22:22:16.310066 22296 leader_election.cc:288] T 00000000000000000000000000000000 P 786730d6bce442ab8790b928b90b94b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:22:16.310120 22306 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 786730d6bce442ab8790b928b90b94b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:22:16.310287 22299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:16.308492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49537 (local address 127.0.98.126:46797)
0618 22:22:16.308604 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:16.308608 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:16.308820 (+ 212us) server_negotiation.cc:407] Connection header received
0618 22:22:16.308974 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:16.308977 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:16.309013 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:16.309119 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:16.309841 (+ 722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:22:16.309845 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:22:16.309847 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:22:16.309858 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:22:16.309882 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:22:16.309893 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:22:16.309996 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0618 22:22:16.310073 (+ 77us) server_negotiation.cc:299] Negotiation successful
0618 22:22:16.310110 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260618 22:22:16.310369 22306 raft_consensus.cc:686] T 00000000000000000000000000000000 P 786730d6bce442ab8790b928b90b94b2 [term 1 LEADER]: Becoming Leader. State: Replica: 786730d6bce442ab8790b928b90b94b2, State: Running, Role: LEADER
I20260618 22:22:16.310633 22306 consensus_queue.cc:237] T 00000000000000000000000000000000 P 786730d6bce442ab8790b928b90b94b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "786730d6bce442ab8790b928b90b94b2" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46797 } }
I20260618 22:22:16.311304 22296 sys_catalog.cc:564] T 00000000000000000000000000000000 P 786730d6bce442ab8790b928b90b94b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:22:16.311817 22306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 786730d6bce442ab8790b928b90b94b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 786730d6bce442ab8790b928b90b94b2. Latest consensus state: current_term: 1 leader_uuid: "786730d6bce442ab8790b928b90b94b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "786730d6bce442ab8790b928b90b94b2" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46797 } } }
I20260618 22:22:16.311990 22306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 786730d6bce442ab8790b928b90b94b2 [sys.catalog]: This master's current role is: LEADER
I20260618 22:22:16.312220 22307 sys_catalog.cc:455] T 00000000000000000000000000000000 P 786730d6bce442ab8790b928b90b94b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "786730d6bce442ab8790b928b90b94b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "786730d6bce442ab8790b928b90b94b2" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 46797 } } }
I20260618 22:22:16.312376 22307 sys_catalog.cc:458] T 00000000000000000000000000000000 P 786730d6bce442ab8790b928b90b94b2 [sys.catalog]: This master's current role is: LEADER
I20260618 22:22:16.312417 22313 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:22:16.315589 22313 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:22:16.321053 22313 catalog_manager.cc:1347] Generated new cluster ID: 74a8fb78ae8b49468c889f0c879c069b
I20260618 22:22:16.321151 22313 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:22:16.330367 22313 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:22:16.330643 22313 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:22:16.335649 22313 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 786730d6bce442ab8790b928b90b94b2: Generated new TSK 0
I20260618 22:22:16.336225 22313 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:22:16.347298 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 22228
2026-06-18T22:22:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1357 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781821065380026-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:22:16 dist-test-slave-g6r4 krb5kdc[22327](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:22:16 dist-test-slave-g6r4 krb5kdc[22327](info): set up 2 sockets
Jun 18 22:22:16 dist-test-slave-g6r4 krb5kdc[22327](info): commencing operation
krb5kdc: starting...
W20260618 22:22:17.422115 393 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781821065380026-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781821065380026-393-0/krb5kdc/test-user.keytab.
Jun 18 22:22:17 dist-test-slave-g6r4 krb5kdc[22327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:22:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:22:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260618 22:22:17.539017 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781821065380026-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781821065380026-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781821065380026-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:37451
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:33781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:37451
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:22:17.643874 22340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:22:17.644178 22340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:22:17.644258 22340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:22:17.648309 22340 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:22:17.648399 22340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:22:17.648439 22340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:22:17.648497 22340 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:22:17.648537 22340 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:22:17.653647 22340 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:33781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781821065380026-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781821065380026-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:37451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:37451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22340
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781821065380026-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:22:17.654695 22340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:22:17.655682 22340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:22:17.663841 22345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:22:17.663859 22346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:22:17.663874 22348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:22:17.664054 22340 server_base.cc:1034] running on GCE node
I20260618 22:22:17.664690 22340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:22:17.665776 22340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:22:17.666944 22340 hybrid_clock.cc:648] HybridClock initialized: now 1781821337666927 us; error 31 us; skew 500 ppm
Jun 18 22:22:17 dist-test-slave-g6r4 krb5kdc[22327](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821337, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:22:17.672564 22340 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260618 22:22:17.673422 22340 webserver.cc:460] Webserver started at http://127.0.98.126:42627/ using document root <none> and password file <none>
I20260618 22:22:17.674050 22340 fs_manager.cc:359] Metadata directory not provided
I20260618 22:22:17.674117 22340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:22:17.674362 22340 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:22:17.676332 22340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781821065380026-393-0/minicluster-data/master-0/data/instance:
uuid: "7aa19a7108a14d378b03c78b1881d24d"
format_stamp: "Formatted at 2026-06-18 22:22:17 on dist-test-slave-g6r4"
server_key: "1a9c01eae6f7ef48add273dd65fdc7f7"
server_key_iv: "697881904bc0b14578153303a887fcc5"
server_key_version: "encryptionkey@0"
I20260618 22:22:17.676894 22340 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781821065380026-393-0/minicluster-data/master-0/wal/instance:
uuid: "7aa19a7108a14d378b03c78b1881d24d"
format_stamp: "Formatted at 2026-06-18 22:22:17 on dist-test-slave-g6r4"
server_key: "1a9c01eae6f7ef48add273dd65fdc7f7"
server_key_iv: "697881904bc0b14578153303a887fcc5"
server_key_version: "encryptionkey@0"
I20260618 22:22:17.680518 22340 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260618 22:22:17.682989 22355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:22:17.684211 22340 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:22:17.684376 22340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781821065380026-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781821065380026-393-0/minicluster-data/master-0/wal
uuid: "7aa19a7108a14d378b03c78b1881d24d"
format_stamp: "Formatted at 2026-06-18 22:22:17 on dist-test-slave-g6r4"
server_key: "1a9c01eae6f7ef48add273dd65fdc7f7"
server_key_iv: "697881904bc0b14578153303a887fcc5"
server_key_version: "encryptionkey@0"
I20260618 22:22:17.684492 22340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781821065380026-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781821065380026-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781821065380026-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:22:17.701193 22340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:22:17.704103 22340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:22:17.704298 22340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:22:17.711796 22340 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:37451
I20260618 22:22:17.711814 22407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:37451 every 8 connection(s)
I20260618 22:22:17.712708 22340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
I20260618 22:22:17.714679 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 22340
I20260618 22:22:17.714816 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781821065380026-393-0/minicluster-data/master-0/wal/instance
I20260618 22:22:17.715133 393 external_mini_cluster.cc:1428] Setting key 30b62bc0ccddc56287f859f74fd7eddd
I20260618 22:22:17.715605 22408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:22:17.722208 22408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7aa19a7108a14d378b03c78b1881d24d: Bootstrap starting.
Jun 18 22:22:17 dist-test-slave-g6r4 krb5kdc[22327](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:22:17.724644 22408 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7aa19a7108a14d378b03c78b1881d24d: Neither blocks nor log segments found. Creating new log.
I20260618 22:22:17.725327 22408 log.cc:824] T 00000000000000000000000000000000 P 7aa19a7108a14d378b03c78b1881d24d: Log is configured to *not* fsync() on all Append() calls
I20260618 22:22:17.727514 22408 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7aa19a7108a14d378b03c78b1881d24d: No bootstrap required, opened a new log
I20260618 22:22:17.729216 22411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:17.716463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48511 (local address 127.0.98.126:37451)
0618 22:22:17.716839 (+ 376us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:17.716860 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:17.716894 (+ 34us) server_negotiation.cc:407] Connection header received
0618 22:22:17.717510 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:17.717533 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:17.717851 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:17.718181 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:17.719080 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:17.720046 (+ 966us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:17.721241 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:17.721696 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:17.721805 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:17.724613 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:22:17.724618 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:22:17.724630 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:22:17.724655 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:22:17.726494 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:17.726983 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:17.726987 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:17.726991 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:17.727093 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:17.727372 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:17.727374 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:17.727375 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:17.727792 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:22:17.727932 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0618 22:22:17.728245 (+ 313us) server_negotiation.cc:299] Negotiation successful
0618 22:22:17.728548 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":127,"threads_started":1}
I20260618 22:22:17.731506 22408 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7aa19a7108a14d378b03c78b1881d24d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aa19a7108a14d378b03c78b1881d24d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 37451 } }
I20260618 22:22:17.731781 22408 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7aa19a7108a14d378b03c78b1881d24d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:22:17.731843 22408 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7aa19a7108a14d378b03c78b1881d24d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7aa19a7108a14d378b03c78b1881d24d, State: Initialized, Role: FOLLOWER
I20260618 22:22:17.732121 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:22:17.732321 22408 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7aa19a7108a14d378b03c78b1881d24d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aa19a7108a14d378b03c78b1881d24d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 37451 } }
I20260618 22:22:17.732467 22408 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7aa19a7108a14d378b03c78b1881d24d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:22:17.732545 22408 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7aa19a7108a14d378b03c78b1881d24d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:22:17.732642 22408 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7aa19a7108a14d378b03c78b1881d24d [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:22:17.733569 22408 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7aa19a7108a14d378b03c78b1881d24d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aa19a7108a14d378b03c78b1881d24d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 37451 } }
I20260618 22:22:17.733951 22408 leader_election.cc:302] T 00000000000000000000000000000000 P 7aa19a7108a14d378b03c78b1881d24d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7aa19a7108a14d378b03c78b1881d24d; no voters:
I20260618 22:22:17.734443 22408 leader_election.cc:288] T 00000000000000000000000000000000 P 7aa19a7108a14d378b03c78b1881d24d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:22:17.734817 22419 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7aa19a7108a14d378b03c78b1881d24d [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:22:17.735097 22419 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7aa19a7108a14d378b03c78b1881d24d [term 1 LEADER]: Becoming Leader. State: Replica: 7aa19a7108a14d378b03c78b1881d24d, State: Running, Role: LEADER
I20260618 22:22:17.735424 22419 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7aa19a7108a14d378b03c78b1881d24d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aa19a7108a14d378b03c78b1881d24d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 37451 } }
I20260618 22:22:17.735921 22408 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7aa19a7108a14d378b03c78b1881d24d [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:22:17.736575 22419 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7aa19a7108a14d378b03c78b1881d24d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7aa19a7108a14d378b03c78b1881d24d. Latest consensus state: current_term: 1 leader_uuid: "7aa19a7108a14d378b03c78b1881d24d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aa19a7108a14d378b03c78b1881d24d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 37451 } } }
I20260618 22:22:17.736754 22419 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7aa19a7108a14d378b03c78b1881d24d [sys.catalog]: This master's current role is: LEADER
I20260618 22:22:17.737085 22420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7aa19a7108a14d378b03c78b1881d24d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7aa19a7108a14d378b03c78b1881d24d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7aa19a7108a14d378b03c78b1881d24d" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 37451 } } }
I20260618 22:22:17.737205 22420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7aa19a7108a14d378b03c78b1881d24d [sys.catalog]: This master's current role is: LEADER
I20260618 22:22:17.737246 22427 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:22:17.741261 22427 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:22:17.743525 22411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:17.734314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58577 (local address 127.0.98.126:37451)
0618 22:22:17.734436 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:17.734441 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:17.734609 (+ 168us) server_negotiation.cc:407] Connection header received
0618 22:22:17.734764 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:17.734767 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:17.734815 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:17.734902 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:17.735742 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:17.736503 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:17.737704 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:17.738039 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:17.738136 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:17.739746 (+ 1610us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:22:17.739750 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:22:17.739753 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:22:17.739774 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:22:17.741424 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:17.741925 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:17.741928 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:17.741930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:17.742023 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:17.742345 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:17.742349 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:17.742351 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:17.742553 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:22:17.742672 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0618 22:22:17.743270 (+ 598us) server_negotiation.cc:299] Negotiation successful
0618 22:22:17.743374 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260618 22:22:17.746852 22427 catalog_manager.cc:1347] Generated new cluster ID: a69a8accf960464ea14760d5fb080a90
I20260618 22:22:17.746912 22427 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:22:17.770961 22427 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:22:17.771821 22427 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:22:17.778280 22427 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7aa19a7108a14d378b03c78b1881d24d: Generated new TSK 0
I20260618 22:22:17.778920 22427 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:22:17.783481 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 22340
2026-06-18T22:22:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1436 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781821065380026-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:22:17 dist-test-slave-g6r4 krb5kdc[22439](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:22:17 dist-test-slave-g6r4 krb5kdc[22439](info): set up 2 sockets
Jun 18 22:22:17 dist-test-slave-g6r4 krb5kdc[22439](info): commencing operation
krb5kdc: starting...
W20260618 22:22:18.826160 393 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781821065380026-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781821065380026-393-0/krb5kdc/test-user.keytab.
Jun 18 22:22:18 dist-test-slave-g6r4 krb5kdc[22439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821338, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:22:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:22:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260618 22:22:18.944445 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781821065380026-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781821065380026-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781821065380026-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:35173
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:40175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:35173
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:22:19.048959 22452 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:22:19.049248 22452 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:22:19.049324 22452 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:22:19.053259 22452 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:22:19.053346 22452 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:22:19.053390 22452 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:22:19.053440 22452 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:22:19.053476 22452 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:22:19.058535 22452 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:40175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781821065380026-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781821065380026-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:35173
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:35173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22452
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781821065380026-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:22:19.059527 22452 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:22:19.060490 22452 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:22:19.068650 22457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:22:19.068640 22458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:22:19.068639 22460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:22:19.068918 22452 server_base.cc:1034] running on GCE node
I20260618 22:22:19.069548 22452 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:22:19.070734 22452 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:22:19.071919 22452 hybrid_clock.cc:648] HybridClock initialized: now 1781821339071901 us; error 32 us; skew 500 ppm
Jun 18 22:22:19 dist-test-slave-g6r4 krb5kdc[22439](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821339, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:22:19.074771 22452 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260618 22:22:19.075618 22452 webserver.cc:460] Webserver started at http://127.0.98.126:44711/ using document root <none> and password file <none>
I20260618 22:22:19.076164 22452 fs_manager.cc:359] Metadata directory not provided
I20260618 22:22:19.076251 22452 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:22:19.076467 22452 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:22:19.078505 22452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781821065380026-393-0/minicluster-data/master-0/data/instance:
uuid: "a575f14959504fab8c752a345932c1d5"
format_stamp: "Formatted at 2026-06-18 22:22:19 on dist-test-slave-g6r4"
server_key: "96a23d5c0fed4e608b42c7f1b78fbd5d"
server_key_iv: "41a52c8e08331d73aae056505b6fb885"
server_key_version: "encryptionkey@0"
I20260618 22:22:19.079113 22452 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781821065380026-393-0/minicluster-data/master-0/wal/instance:
uuid: "a575f14959504fab8c752a345932c1d5"
format_stamp: "Formatted at 2026-06-18 22:22:19 on dist-test-slave-g6r4"
server_key: "96a23d5c0fed4e608b42c7f1b78fbd5d"
server_key_iv: "41a52c8e08331d73aae056505b6fb885"
server_key_version: "encryptionkey@0"
I20260618 22:22:19.082706 22452 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260618 22:22:19.084972 22467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:22:19.086205 22452 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:22:19.086329 22452 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781821065380026-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781821065380026-393-0/minicluster-data/master-0/wal
uuid: "a575f14959504fab8c752a345932c1d5"
format_stamp: "Formatted at 2026-06-18 22:22:19 on dist-test-slave-g6r4"
server_key: "96a23d5c0fed4e608b42c7f1b78fbd5d"
server_key_iv: "41a52c8e08331d73aae056505b6fb885"
server_key_version: "encryptionkey@0"
I20260618 22:22:19.086431 22452 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781821065380026-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781821065380026-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781821065380026-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:22:19.104470 22452 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:22:19.107282 22452 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:22:19.107472 22452 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:22:19.114225 22452 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:35173
I20260618 22:22:19.114238 22519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:35173 every 8 connection(s)
I20260618 22:22:19.115128 22452 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
I20260618 22:22:19.118069 22520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:22:19.119632 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 22452
I20260618 22:22:19.119784 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781821065380026-393-0/minicluster-data/master-0/wal/instance
I20260618 22:22:19.120047 393 external_mini_cluster.cc:1428] Setting key bc88177625c7644aa168eddb9da59777
I20260618 22:22:19.124169 22520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a575f14959504fab8c752a345932c1d5: Bootstrap starting.
I20260618 22:22:19.126361 22520 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a575f14959504fab8c752a345932c1d5: Neither blocks nor log segments found. Creating new log.
I20260618 22:22:19.127059 22520 log.cc:824] T 00000000000000000000000000000000 P a575f14959504fab8c752a345932c1d5: Log is configured to *not* fsync() on all Append() calls
I20260618 22:22:19.128597 22523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:19.121343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58047 (local address 127.0.98.126:35173)
0618 22:22:19.121759 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:19.121773 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:19.121807 (+ 34us) server_negotiation.cc:407] Connection header received
0618 22:22:19.122451 (+ 644us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:19.122466 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:19.122796 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:19.123114 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:19.123511 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:19.124475 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:19.125391 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:19.125859 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:19.126067 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:19.126890 (+ 823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:22:19.126901 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:22:19.126914 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:22:19.127357 (+ 443us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:22:19.127431 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:22:19.127468 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:22:19.127579 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0618 22:22:19.127738 (+ 159us) server_negotiation.cc:299] Negotiation successful
0618 22:22:19.127931 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":125,"threads_started":1}
I20260618 22:22:19.129161 22520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a575f14959504fab8c752a345932c1d5: No bootstrap required, opened a new log
I20260618 22:22:19.131301 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:22:19.133289 22520 raft_consensus.cc:348] T 00000000000000000000000000000000 P a575f14959504fab8c752a345932c1d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a575f14959504fab8c752a345932c1d5" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 35173 } }
I20260618 22:22:19.133545 22520 raft_consensus.cc:374] T 00000000000000000000000000000000 P a575f14959504fab8c752a345932c1d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:22:19.133621 22520 raft_consensus.cc:729] T 00000000000000000000000000000000 P a575f14959504fab8c752a345932c1d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a575f14959504fab8c752a345932c1d5, State: Initialized, Role: FOLLOWER
I20260618 22:22:19.134130 22520 consensus_queue.cc:260] T 00000000000000000000000000000000 P a575f14959504fab8c752a345932c1d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a575f14959504fab8c752a345932c1d5" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 35173 } }
I20260618 22:22:19.134259 22520 raft_consensus.cc:388] T 00000000000000000000000000000000 P a575f14959504fab8c752a345932c1d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:22:19.134366 22520 raft_consensus.cc:482] T 00000000000000000000000000000000 P a575f14959504fab8c752a345932c1d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:22:19.134479 22520 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a575f14959504fab8c752a345932c1d5 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:22:19.135640 22520 raft_consensus.cc:504] T 00000000000000000000000000000000 P a575f14959504fab8c752a345932c1d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a575f14959504fab8c752a345932c1d5" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 35173 } }
I20260618 22:22:19.135988 22520 leader_election.cc:302] T 00000000000000000000000000000000 P a575f14959504fab8c752a345932c1d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a575f14959504fab8c752a345932c1d5; no voters:
I20260618 22:22:19.136354 22520 leader_election.cc:288] T 00000000000000000000000000000000 P a575f14959504fab8c752a345932c1d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:22:19.136369 22530 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a575f14959504fab8c752a345932c1d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:22:19.136581 22530 raft_consensus.cc:686] T 00000000000000000000000000000000 P a575f14959504fab8c752a345932c1d5 [term 1 LEADER]: Becoming Leader. State: Replica: a575f14959504fab8c752a345932c1d5, State: Running, Role: LEADER
I20260618 22:22:19.136976 22530 consensus_queue.cc:237] T 00000000000000000000000000000000 P a575f14959504fab8c752a345932c1d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a575f14959504fab8c752a345932c1d5" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 35173 } }
I20260618 22:22:19.137584 22523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:19.133147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58591 (local address 127.0.98.126:35173)
0618 22:22:19.133250 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:19.133255 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:19.133421 (+ 166us) server_negotiation.cc:407] Connection header received
0618 22:22:19.133563 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:19.133566 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:19.133610 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:19.133694 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:19.134041 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:19.134697 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:19.135799 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:19.136054 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:19.136168 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:19.137137 (+ 969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:22:19.137140 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:22:19.137143 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:22:19.137155 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:22:19.137182 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:22:19.137194 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:22:19.137234 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0618 22:22:19.137427 (+ 193us) server_negotiation.cc:299] Negotiation successful
0618 22:22:19.137477 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260618 22:22:19.137696 22520 sys_catalog.cc:564] T 00000000000000000000000000000000 P a575f14959504fab8c752a345932c1d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:22:19.139050 22531 sys_catalog.cc:455] T 00000000000000000000000000000000 P a575f14959504fab8c752a345932c1d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a575f14959504fab8c752a345932c1d5. Latest consensus state: current_term: 1 leader_uuid: "a575f14959504fab8c752a345932c1d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a575f14959504fab8c752a345932c1d5" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 35173 } } }
I20260618 22:22:19.139187 22531 sys_catalog.cc:458] T 00000000000000000000000000000000 P a575f14959504fab8c752a345932c1d5 [sys.catalog]: This master's current role is: LEADER
I20260618 22:22:19.139453 22531 sys_catalog.cc:455] T 00000000000000000000000000000000 P a575f14959504fab8c752a345932c1d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a575f14959504fab8c752a345932c1d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a575f14959504fab8c752a345932c1d5" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 35173 } } }
I20260618 22:22:19.139551 22531 sys_catalog.cc:458] T 00000000000000000000000000000000 P a575f14959504fab8c752a345932c1d5 [sys.catalog]: This master's current role is: LEADER
I20260618 22:22:19.139608 22539 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:22:19.142213 22539 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:22:19.147192 22539 catalog_manager.cc:1347] Generated new cluster ID: 2bb86fcc2fc142bc80105c945dd28203
I20260618 22:22:19.147253 22539 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:22:19.161746 22539 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:22:19.162694 22539 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:22:19.170941 22539 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a575f14959504fab8c752a345932c1d5: Generated new TSK 0
I20260618 22:22:19.171643 22539 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:22:19.175346 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 22452
2026-06-18T22:22:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1391 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781821065380026-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:22:19 dist-test-slave-g6r4 krb5kdc[22551](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:22:19 dist-test-slave-g6r4 krb5kdc[22551](info): set up 2 sockets
Jun 18 22:22:19 dist-test-slave-g6r4 krb5kdc[22551](info): commencing operation
krb5kdc: starting...
W20260618 22:22:20.246181 393 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781821065380026-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781821065380026-393-0/krb5kdc/test-user.keytab.
Jun 18 22:22:20 dist-test-slave-g6r4 krb5kdc[22551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:22:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:22:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260618 22:22:20.362785 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781821065380026-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781821065380026-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781821065380026-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:36761
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:35609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:36761
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:22:20.470093 22564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:22:20.470332 22564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:22:20.470372 22564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:22:20.474253 22564 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:22:20.474298 22564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:22:20.474318 22564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:22:20.474340 22564 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:22:20.474355 22564 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:22:20.479241 22564 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:35609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781821065380026-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781821065380026-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:36761
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:36761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781821065380026-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:22:20.480122 22564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:22:20.481014 22564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:22:20.489086 22572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:22:20.489120 22570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:22:20.489113 22569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:22:20.489356 22564 server_base.cc:1034] running on GCE node
I20260618 22:22:20.490010 22564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:22:20.491112 22564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:22:20.492274 22564 hybrid_clock.cc:648] HybridClock initialized: now 1781821340492256 us; error 30 us; skew 500 ppm
Jun 18 22:22:20 dist-test-slave-g6r4 krb5kdc[22551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821340, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:22:20.494907 22564 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260618 22:22:20.495723 22564 webserver.cc:460] Webserver started at http://127.0.98.126:46401/ using document root <none> and password file <none>
I20260618 22:22:20.496284 22564 fs_manager.cc:359] Metadata directory not provided
I20260618 22:22:20.496368 22564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:22:20.496577 22564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:22:20.498576 22564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781821065380026-393-0/minicluster-data/master-0/data/instance:
uuid: "c96cef15138e424f866e0374f286364c"
format_stamp: "Formatted at 2026-06-18 22:22:20 on dist-test-slave-g6r4"
server_key: "28f6fec2b40e357cdf921854576cbb37"
server_key_iv: "2b5a4fa2d019ee2b732f4a57a81a599a"
server_key_version: "encryptionkey@0"
I20260618 22:22:20.499114 22564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781821065380026-393-0/minicluster-data/master-0/wal/instance:
uuid: "c96cef15138e424f866e0374f286364c"
format_stamp: "Formatted at 2026-06-18 22:22:20 on dist-test-slave-g6r4"
server_key: "28f6fec2b40e357cdf921854576cbb37"
server_key_iv: "2b5a4fa2d019ee2b732f4a57a81a599a"
server_key_version: "encryptionkey@0"
I20260618 22:22:20.502501 22564 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260618 22:22:20.504678 22579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:22:20.505810 22564 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:22:20.505964 22564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781821065380026-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781821065380026-393-0/minicluster-data/master-0/wal
uuid: "c96cef15138e424f866e0374f286364c"
format_stamp: "Formatted at 2026-06-18 22:22:20 on dist-test-slave-g6r4"
server_key: "28f6fec2b40e357cdf921854576cbb37"
server_key_iv: "2b5a4fa2d019ee2b732f4a57a81a599a"
server_key_version: "encryptionkey@0"
I20260618 22:22:20.506067 22564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781821065380026-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781821065380026-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781821065380026-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:22:20.510324 22564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:22:20.512876 22564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:22:20.513057 22564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:22:20.519609 22564 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:36761
I20260618 22:22:20.519639 22631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:36761 every 8 connection(s)
I20260618 22:22:20.520439 22564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
I20260618 22:22:20.522909 22632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:22:20.527900 22632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c96cef15138e424f866e0374f286364c: Bootstrap starting.
I20260618 22:22:20.528123 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 22564
I20260618 22:22:20.528255 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781821065380026-393-0/minicluster-data/master-0/wal/instance
I20260618 22:22:20.528540 393 external_mini_cluster.cc:1428] Setting key 02dcd4e89e241f56f5b8327e7d46911d
I20260618 22:22:20.530416 22632 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c96cef15138e424f866e0374f286364c: Neither blocks nor log segments found. Creating new log.
I20260618 22:22:20.531240 22632 log.cc:824] T 00000000000000000000000000000000 P c96cef15138e424f866e0374f286364c: Log is configured to *not* fsync() on all Append() calls
I20260618 22:22:20.533082 22632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c96cef15138e424f866e0374f286364c: No bootstrap required, opened a new log
I20260618 22:22:20.533768 22635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:20.529806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37927 (local address 127.0.98.126:36761)
0618 22:22:20.530261 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:20.530281 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:20.530321 (+ 40us) server_negotiation.cc:407] Connection header received
0618 22:22:20.531164 (+ 843us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:20.531187 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:20.531528 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:20.531877 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:20.532633 (+ 756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:22:20.532639 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:22:20.532654 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:22:20.532669 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:22:20.532718 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:22:20.532737 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:22:20.532844 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0618 22:22:20.532951 (+ 107us) server_negotiation.cc:299] Negotiation successful
0618 22:22:20.533100 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":115,"threads_started":1}
I20260618 22:22:20.536317 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:22:20.536571 22632 raft_consensus.cc:348] T 00000000000000000000000000000000 P c96cef15138e424f866e0374f286364c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c96cef15138e424f866e0374f286364c" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36761 } }
I20260618 22:22:20.536760 22632 raft_consensus.cc:374] T 00000000000000000000000000000000 P c96cef15138e424f866e0374f286364c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 22:22:20.536854 22632 raft_consensus.cc:729] T 00000000000000000000000000000000 P c96cef15138e424f866e0374f286364c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c96cef15138e424f866e0374f286364c, State: Initialized, Role: FOLLOWER
I20260618 22:22:20.537264 22632 consensus_queue.cc:260] T 00000000000000000000000000000000 P c96cef15138e424f866e0374f286364c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c96cef15138e424f866e0374f286364c" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36761 } }
I20260618 22:22:20.537395 22632 raft_consensus.cc:388] T 00000000000000000000000000000000 P c96cef15138e424f866e0374f286364c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 22:22:20.537467 22632 raft_consensus.cc:482] T 00000000000000000000000000000000 P c96cef15138e424f866e0374f286364c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 22:22:20.537583 22632 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c96cef15138e424f866e0374f286364c [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:22:20.538597 22632 raft_consensus.cc:504] T 00000000000000000000000000000000 P c96cef15138e424f866e0374f286364c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c96cef15138e424f866e0374f286364c" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36761 } }
I20260618 22:22:20.538973 22632 leader_election.cc:302] T 00000000000000000000000000000000 P c96cef15138e424f866e0374f286364c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c96cef15138e424f866e0374f286364c; no voters:
I20260618 22:22:20.539295 22632 leader_election.cc:288] T 00000000000000000000000000000000 P c96cef15138e424f866e0374f286364c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 22:22:20.539383 22642 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c96cef15138e424f866e0374f286364c [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:22:20.539623 22642 raft_consensus.cc:686] T 00000000000000000000000000000000 P c96cef15138e424f866e0374f286364c [term 1 LEADER]: Becoming Leader. State: Replica: c96cef15138e424f866e0374f286364c, State: Running, Role: LEADER
I20260618 22:22:20.539938 22642 consensus_queue.cc:237] T 00000000000000000000000000000000 P c96cef15138e424f866e0374f286364c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c96cef15138e424f866e0374f286364c" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36761 } }
I20260618 22:22:20.540119 22635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:20.538311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35889 (local address 127.0.98.126:36761)
0618 22:22:20.538415 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:20.538419 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:20.538568 (+ 149us) server_negotiation.cc:407] Connection header received
0618 22:22:20.538720 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:20.538724 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:20.538757 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:20.538828 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:20.539580 (+ 752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:22:20.539584 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:22:20.539587 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 22:22:20.539597 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:22:20.539621 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 22:22:20.539633 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:22:20.539703 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0618 22:22:20.539983 (+ 280us) server_negotiation.cc:299] Negotiation successful
0618 22:22:20.540017 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260618 22:22:20.540585 22632 sys_catalog.cc:564] T 00000000000000000000000000000000 P c96cef15138e424f866e0374f286364c [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:22:20.541735 22644 sys_catalog.cc:455] T 00000000000000000000000000000000 P c96cef15138e424f866e0374f286364c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c96cef15138e424f866e0374f286364c. Latest consensus state: current_term: 1 leader_uuid: "c96cef15138e424f866e0374f286364c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c96cef15138e424f866e0374f286364c" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36761 } } }
I20260618 22:22:20.541857 22644 sys_catalog.cc:458] T 00000000000000000000000000000000 P c96cef15138e424f866e0374f286364c [sys.catalog]: This master's current role is: LEADER
I20260618 22:22:20.542192 22643 sys_catalog.cc:455] T 00000000000000000000000000000000 P c96cef15138e424f866e0374f286364c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c96cef15138e424f866e0374f286364c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c96cef15138e424f866e0374f286364c" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36761 } } }
I20260618 22:22:20.542268 22643 sys_catalog.cc:458] T 00000000000000000000000000000000 P c96cef15138e424f866e0374f286364c [sys.catalog]: This master's current role is: LEADER
I20260618 22:22:20.542269 22651 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:22:20.545258 22651 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:22:20.549844 22651 catalog_manager.cc:1347] Generated new cluster ID: 2df454b02dc54ccbb81056ae57bd9848
I20260618 22:22:20.549921 22651 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:22:20.579758 22651 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:22:20.580086 22651 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:22:20.584626 22651 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c96cef15138e424f866e0374f286364c: Generated new TSK 0
I20260618 22:22:20.585227 22651 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 22:22:20.598452 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 22564
2026-06-18T22:22:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1423 ms)
[----------] 18 tests from AuthTokenIssuingTest (25138 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821065380026-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:22:20 dist-test-slave-g6r4 krb5kdc[22663](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:22:20 dist-test-slave-g6r4 krb5kdc[22663](info): set up 2 sockets
Jun 18 22:22:20 dist-test-slave-g6r4 krb5kdc[22663](info): commencing operation
krb5kdc: starting...
W20260618 22:22:21.678326 393 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821065380026-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821065380026-393-0/krb5kdc/test-user.keytab.
Jun 18 22:22:21 dist-test-slave-g6r4 krb5kdc[22663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821341, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:22:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:22:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260618 22:22:21.794966 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821065380026-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821065380026-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821065380026-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:43549
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:43985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:43549,127.0.98.125:33567,127.0.98.124:35989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:22:21.902586 22676 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260618 22:22:21.902878 22676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:22:21.902945 22676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:22:21.903023 22676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:22:21.906863 22676 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:22:21.906941 22676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:22:21.906978 22676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:22:21.907009 22676 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:22:21.907047 22676 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:22:21.912160 22676 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:43985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821065380026-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821065380026-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:43549,127.0.98.125:33567,127.0.98.124:35989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:43549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22676
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821065380026-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:22:21.913144 22676 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:22:21.914393 22676 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:22:21.922709 22682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:22:21.922832 22676 server_base.cc:1034] running on GCE node
W20260618 22:22:21.922721 22681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:22:21.922739 22684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:22:21.923450 22676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:22:21.924538 22676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:22:21.925711 22676 hybrid_clock.cc:648] HybridClock initialized: now 1781821341925685 us; error 38 us; skew 500 ppm
Jun 18 22:22:21 dist-test-slave-g6r4 krb5kdc[22663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821341, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:22:21.928462 22676 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260618 22:22:21.929226 22676 webserver.cc:460] Webserver started at http://127.0.98.126:33781/ using document root <none> and password file <none>
I20260618 22:22:21.929821 22676 fs_manager.cc:359] Metadata directory not provided
I20260618 22:22:21.929883 22676 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:22:21.930095 22676 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:22:21.931990 22676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821065380026-393-0/minicluster-data/master-0/data/instance:
uuid: "1116e0c1f1d046b3bac47ad94de725a6"
format_stamp: "Formatted at 2026-06-18 22:22:21 on dist-test-slave-g6r4"
server_key: "400a52b7f6c0577cd0fd81d44e73c8db"
server_key_iv: "29481905fe689b2b271ccd57ebb91b10"
server_key_version: "encryptionkey@0"
I20260618 22:22:21.932497 22676 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821065380026-393-0/minicluster-data/master-0/wal/instance:
uuid: "1116e0c1f1d046b3bac47ad94de725a6"
format_stamp: "Formatted at 2026-06-18 22:22:21 on dist-test-slave-g6r4"
server_key: "400a52b7f6c0577cd0fd81d44e73c8db"
server_key_iv: "29481905fe689b2b271ccd57ebb91b10"
server_key_version: "encryptionkey@0"
I20260618 22:22:21.935961 22676 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260618 22:22:21.938639 22691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:22:21.939819 22676 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:22:21.939956 22676 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821065380026-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821065380026-393-0/minicluster-data/master-0/wal
uuid: "1116e0c1f1d046b3bac47ad94de725a6"
format_stamp: "Formatted at 2026-06-18 22:22:21 on dist-test-slave-g6r4"
server_key: "400a52b7f6c0577cd0fd81d44e73c8db"
server_key_iv: "29481905fe689b2b271ccd57ebb91b10"
server_key_version: "encryptionkey@0"
I20260618 22:22:21.940048 22676 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821065380026-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821065380026-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821065380026-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:22:21.962169 22676 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:22:21.965277 22676 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:22:21.965483 22676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:22:21.972393 22676 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:43549
I20260618 22:22:21.972656 22743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:43549 every 8 connection(s)
I20260618 22:22:21.973380 22676 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
I20260618 22:22:21.976323 22744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:22:21.979934 22744 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.126" port: 43549 } has no permanent_uuid. Determining permanent_uuid...
I20260618 22:22:21.980597 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 22676
I20260618 22:22:21.980755 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821065380026-393-0/minicluster-data/master-0/wal/instance
I20260618 22:22:21.981033 393 external_mini_cluster.cc:1428] Setting key 6a20789ddcea7d56fad7abfe6459e2f1
WARNING: no policy specified for kudu/127.0.98.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.125@KRBTEST.COM" created.
Jun 18 22:22:21 dist-test-slave-g6r4 krb5kdc[22663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821341, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:22:21.999317 22748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:21.982274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:43549 (local address 127.0.0.1:55492)
0618 22:22:21.982879 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0618 22:22:21.982934 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0618 22:22:21.984091 (+ 1157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:22:21.984914 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:22:21.984919 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:22:21.985280 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:22:21.985541 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:22:21.985554 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:22:21.987390 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:22:21.987393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:22:21.988102 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:22:21.988111 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:22:21.988983 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:22:21.988986 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:22:21.989083 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:21.989696 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:22:21.989711 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:22:21.993703 (+ 3992us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:22:21.996360 (+ 2657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:22:21.996367 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:22:21.996375 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:22:21.996634 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:22:21.997058 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:22:21.997060 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:22:21.997062 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:22:21.997162 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:22:21.998037 (+ 875us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:22:21.998042 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:22:21.998158 (+ 116us) client_negotiation.cc:769] Sending connection context
0618 22:22:21.998305 (+ 147us) client_negotiation.cc:240] Negotiation successful
0618 22:22:21.998536 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":403,"spinlock_wait_cycles":42368,"thread_start_us":165,"threads_started":1}
I20260618 22:22:21.999326 22746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:21.982273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55492 (local address 127.0.98.126:43549)
0618 22:22:21.982679 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:21.982698 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:21.983058 (+ 360us) server_negotiation.cc:407] Connection header received
0618 22:22:21.984434 (+ 1376us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:21.984443 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:21.984678 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:21.984875 (+ 197us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:21.985690 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:21.987263 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:21.988224 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:21.988833 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:21.989044 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:21.993859 (+ 4815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:22:21.993865 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:22:21.993875 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:22:21.993939 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:22:21.996248 (+ 2309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:21.996781 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:21.996791 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:21.996795 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:21.996957 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:21.997256 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:21.997263 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:21.997267 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:21.997880 (+ 613us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:22:21.998081 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0618 22:22:21.998436 (+ 355us) server_negotiation.cc:299] Negotiation successful
0618 22:22:21.998650 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":266,"thread_start_us":136,"threads_started":1}
I20260618 22:22:22.001893 22744 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.125" port: 33567 } has no permanent_uuid. Determining permanent_uuid...
I20260618 22:22:22.002631 22748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:22.002403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:33567 (local address 127.0.0.1:44734)
0618 22:22:22.002510 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0618 22:22:22.002575 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.125:33567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260618 22:22:22.002823 22692 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.125:33567: connect: Connection refused (error 111)
I20260618 22:22:22.003609 22748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:22.003430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:33567 (local address 127.0.0.1:44750)
0618 22:22:22.003524 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0618 22:22:22.003565 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.125:33567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260618 22:22:22.003739 22744 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.125:33567: Network error: Client connection negotiation failed: client connection to 127.0.98.125:33567: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821065380026-393-0/krb5kdc/kudu_127.0.98.125.keytab.
Entry for principal kudu/127.0.98.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821065380026-393-0/krb5kdc/kudu_127.0.98.125.keytab.
I20260618 22:22:22.009534 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821065380026-393-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821065380026-393-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821065380026-393-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821065380026-393-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.125:33567
--webserver_interface=127.0.98.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821065380026-393-0/krb5kdc/kudu_127.0.98.125.keytab
--principal=kudu/127.0.98.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:43985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:43549,127.0.98.125:33567,127.0.98.124:35989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 22:22:22.053011 22744 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.125" port: 33567 } attempt: 1
I20260618 22:22:22.053982 22748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:22.053632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:33567 (local address 127.0.0.1:44762)
0618 22:22:22.053783 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0618 22:22:22.053852 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.125:33567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260618 22:22:22.054735 22748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:22.054540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:33567 (local address 127.0.0.1:44778)
0618 22:22:22.054631 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0618 22:22:22.054680 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.125:33567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260618 22:22:22.054880 22744 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.125:33567: Network error: Client connection negotiation failed: client connection to 127.0.98.125:33567: connect: Connection refused (error 111)
W20260618 22:22:22.117249 22751 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260618 22:22:22.117506 22751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:22:22.117528 22751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:22:22.117604 22751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:22:22.121512 22751 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:22:22.121567 22751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:22:22.121585 22751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:22:22.121601 22751 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:22:22.121616 22751 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:22:22.123049 22744 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.125" port: 33567 } attempt: 2
I20260618 22:22:22.123878 22748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:22.123603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:33567 (local address 127.0.0.1:44784)
0618 22:22:22.123741 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0618 22:22:22.123803 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.125:33567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260618 22:22:22.124565 22748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:22.124388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:33567 (local address 127.0.0.1:44800)
0618 22:22:22.124476 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0618 22:22:22.124518 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.125:33567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260618 22:22:22.124742 22744 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.125:33567: Network error: Client connection negotiation failed: client connection to 127.0.98.125:33567: connect: Connection refused (error 111)
I20260618 22:22:22.127370 22751 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:43985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821065380026-393-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821065380026-393-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:43549,127.0.98.125:33567,127.0.98.124:35989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821065380026-393-0/krb5kdc/kudu_127.0.98.125.keytab
--principal=kudu/127.0.98.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.125:33567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821065380026-393-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22751
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821065380026-393-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:22:22.128342 22751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:22:22.129308 22751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:22:22.137616 22756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:22:22.137653 22757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:22:22.137630 22759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:22:22.137677 22751 server_base.cc:1034] running on GCE node
I20260618 22:22:22.138454 22751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:22:22.139535 22751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:22:22.140712 22751 hybrid_clock.cc:648] HybridClock initialized: now 1781821342140701 us; error 31 us; skew 500 ppm
Jun 18 22:22:22 dist-test-slave-g6r4 krb5kdc[22663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821342, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:22:22.143385 22751 init.cc:377] Logged in from keytab as kudu/127.0.98.125@KRBTEST.COM (short username kudu)
I20260618 22:22:22.144255 22751 webserver.cc:460] Webserver started at http://127.0.98.125:39511/ using document root <none> and password file <none>
I20260618 22:22:22.144826 22751 fs_manager.cc:359] Metadata directory not provided
I20260618 22:22:22.144925 22751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:22:22.145095 22751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:22:22.147064 22751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821065380026-393-0/minicluster-data/master-1/data/instance:
uuid: "080ee8a2d01e4028a36ab076df4fbc75"
format_stamp: "Formatted at 2026-06-18 22:22:22 on dist-test-slave-g6r4"
server_key: "9eb141e3fea2e45d284b43541e500542"
server_key_iv: "b084c1da37baace5eeb5a8c97de5e668"
server_key_version: "encryptionkey@0"
I20260618 22:22:22.147554 22751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821065380026-393-0/minicluster-data/master-1/wal/instance:
uuid: "080ee8a2d01e4028a36ab076df4fbc75"
format_stamp: "Formatted at 2026-06-18 22:22:22 on dist-test-slave-g6r4"
server_key: "9eb141e3fea2e45d284b43541e500542"
server_key_iv: "b084c1da37baace5eeb5a8c97de5e668"
server_key_version: "encryptionkey@0"
I20260618 22:22:22.150985 22751 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260618 22:22:22.153210 22766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:22:22.154441 22751 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:22:22.154536 22751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821065380026-393-0/minicluster-data/master-1/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821065380026-393-0/minicluster-data/master-1/wal
uuid: "080ee8a2d01e4028a36ab076df4fbc75"
format_stamp: "Formatted at 2026-06-18 22:22:22 on dist-test-slave-g6r4"
server_key: "9eb141e3fea2e45d284b43541e500542"
server_key_iv: "b084c1da37baace5eeb5a8c97de5e668"
server_key_version: "encryptionkey@0"
I20260618 22:22:22.154616 22751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821065380026-393-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821065380026-393-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821065380026-393-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:22:22.168982 22751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:22:22.171860 22751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:22:22.172065 22751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:22:22.178823 22751 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.125:33567
I20260618 22:22:22.179108 22818 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.125:33567 every 8 connection(s)
I20260618 22:22:22.179775 22751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821065380026-393-0/minicluster-data/master-1/data/info.pb
I20260618 22:22:22.182492 22819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:22:22.185326 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 22751
I20260618 22:22:22.185464 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821065380026-393-0/minicluster-data/master-1/wal/instance
I20260618 22:22:22.185773 393 external_mini_cluster.cc:1428] Setting key b49b6bc9d488ce770261697e347a2f68
I20260618 22:22:22.186411 22819 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.126" port: 43549 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.124@KRBTEST.COM" created.
Jun 18 22:22:22 dist-test-slave-g6r4 krb5kdc[22663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821342, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.125@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:22:22.201032 22746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:22.188545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55496 (local address 127.0.98.126:43549)
0618 22:22:22.188666 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:22.188670 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:22.189624 (+ 954us) server_negotiation.cc:407] Connection header received
0618 22:22:22.190618 (+ 994us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:22.190622 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:22.190683 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:22.190785 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:22.191897 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:22.192444 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:22.193571 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:22.193810 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:22.193931 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:22.196887 (+ 2956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:22:22.196894 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:22:22.196901 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:22:22.196935 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:22:22.198567 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:22.199067 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:22.199074 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:22.199079 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:22.199173 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:22.199480 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:22.199486 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:22.199491 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:22.199746 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:22:22.199873 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0618 22:22:22.200808 (+ 935us) server_negotiation.cc:299] Negotiation successful
0618 22:22:22.200907 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260618 22:22:22.201241 22822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:22.189005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:43549 (local address 127.0.0.1:55496)
0618 22:22:22.189473 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0618 22:22:22.189512 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0618 22:22:22.190428 (+ 916us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:22:22.190954 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:22:22.190961 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:22:22.191300 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:22:22.191743 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:22:22.191755 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:22:22.192567 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:22:22.192575 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:22:22.193455 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:22:22.193465 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:22:22.193981 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:22:22.193985 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:22:22.194139 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:22.194698 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:22:22.194713 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:22:22.196716 (+ 2003us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:22:22.198671 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:22:22.198675 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:22:22.198685 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:22:22.198963 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:22:22.199265 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:22:22.199267 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:22:22.199268 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:22:22.199387 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:22:22.199859 (+ 472us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:22:22.199864 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:22:22.200096 (+ 232us) client_negotiation.cc:769] Sending connection context
0618 22:22:22.200248 (+ 152us) client_negotiation.cc:240] Negotiation successful
0618 22:22:22.200469 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":118,"threads_started":1}
I20260618 22:22:22.202798 22819 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.125" port: 33567 } has no permanent_uuid. Determining permanent_uuid...
Jun 18 22:22:22 dist-test-slave-g6r4 krb5kdc[22663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821342, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.125@KRBTEST.COM for kudu/127.0.98.125@KRBTEST.COM
I20260618 22:22:22.211939 22822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:22.203255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:33567 (local address 127.0.0.1:44806)
0618 22:22:22.203366 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0618 22:22:22.203381 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:22:22.203466 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:22:22.204148 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:22:22.204151 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:22:22.204277 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:22:22.204363 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:22:22.204368 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:22:22.205627 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:22:22.205629 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:22:22.206290 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:22:22.206299 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:22:22.206850 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:22:22.206853 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:22:22.206934 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:22.207443 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:22:22.207454 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:22:22.208617 (+ 1163us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:22:22.210478 (+ 1861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:22:22.210481 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:22:22.210502 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:22:22.210681 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:22:22.211007 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:22:22.211010 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:22:22.211012 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:22:22.211095 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:22:22.211514 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:22:22.211516 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:22:22.211588 (+ 72us) client_negotiation.cc:769] Sending connection context
0618 22:22:22.211687 (+ 99us) client_negotiation.cc:240] Negotiation successful
0618 22:22:22.211815 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260618 22:22:22.212095 22824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:22.203412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44806 (local address 127.0.98.125:33567)
0618 22:22:22.203743 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:22.203756 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:22.203777 (+ 21us) server_negotiation.cc:407] Connection header received
0618 22:22:22.203821 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:22.203825 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:22.204016 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:22.204164 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:22.204476 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:22.205526 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:22.206411 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:22.206756 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:22.206844 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:22.208733 (+ 1889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:22:22.208741 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:22:22.208750 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:22:22.208770 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:22:22.210385 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:22.210805 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:22.210810 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:22.210811 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:22.210896 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:22.211173 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:22.211176 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:22.211177 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:22.211414 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:22:22.211519 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0618 22:22:22.211754 (+ 235us) server_negotiation.cc:299] Negotiation successful
0618 22:22:22.211875 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":144,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821065380026-393-0/krb5kdc/kudu_127.0.98.124.keytab.
Entry for principal kudu/127.0.98.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821065380026-393-0/krb5kdc/kudu_127.0.98.124.keytab.
I20260618 22:22:22.213373 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821065380026-393-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821065380026-393-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821065380026-393-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821065380026-393-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.124:35989
--webserver_interface=127.0.98.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821065380026-393-0/krb5kdc/kudu_127.0.98.124.keytab
--principal=kudu/127.0.98.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:43985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:43549,127.0.98.125:33567,127.0.98.124:35989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 22:22:22.213879 22819 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.124" port: 35989 } has no permanent_uuid. Determining permanent_uuid...
I20260618 22:22:22.214668 22822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:22.214376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:35989 (local address 127.0.0.1:34826)
0618 22:22:22.214527 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0618 22:22:22.214606 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:35989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260618 22:22:22.214838 22767 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.124:35989: connect: Connection refused (error 111)
I20260618 22:22:22.215590 22822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:22.215394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:35989 (local address 127.0.0.1:34838)
0618 22:22:22.215494 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0618 22:22:22.215538 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:35989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260618 22:22:22.215735 22819 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.124:35989: Network error: Client connection negotiation failed: client connection to 127.0.98.124:35989: connect: Connection refused (error 111)
I20260618 22:22:22.215886 22744 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.125" port: 33567 } attempt: 3
Jun 18 22:22:22 dist-test-slave-g6r4 krb5kdc[22663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821341, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for kudu/127.0.98.125@KRBTEST.COM
I20260618 22:22:22.225248 22824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:22.216386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44818 (local address 127.0.98.125:33567)
0618 22:22:22.216728 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:22.216735 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:22.216750 (+ 15us) server_negotiation.cc:407] Connection header received
0618 22:22:22.216789 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:22.216796 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:22.216851 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:22.216943 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:22.217399 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:22.218423 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:22.219186 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:22.219690 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:22.219831 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:22.222221 (+ 2390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:22:22.222224 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:22:22.222226 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:22:22.222246 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:22:22.223617 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:22.224152 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:22.224154 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:22.224155 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:22.224221 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:22.224532 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:22.224534 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:22.224535 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:22.224675 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:22:22.224744 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0618 22:22:22.225076 (+ 332us) server_negotiation.cc:299] Negotiation successful
0618 22:22:22.225158 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":250}
I20260618 22:22:22.225477 22748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:22.216278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:33567 (local address 127.0.0.1:44818)
0618 22:22:22.216444 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0618 22:22:22.216463 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0618 22:22:22.216553 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:22:22.216973 (+ 420us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:22:22.216975 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:22:22.217153 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:22:22.217257 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:22:22.217263 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:22:22.218533 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:22:22.218536 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:22:22.219073 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:22:22.219080 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:22:22.219807 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:22:22.219809 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:22:22.219889 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:22.220422 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:22:22.220434 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:22:22.222084 (+ 1650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:22:22.223728 (+ 1644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:22:22.223735 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:22:22.223738 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:22:22.224048 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:22:22.224311 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:22:22.224317 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:22:22.224322 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:22:22.224437 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:22:22.224767 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:22:22.224773 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:22:22.224905 (+ 132us) client_negotiation.cc:769] Sending connection context
0618 22:22:22.225044 (+ 139us) client_negotiation.cc:240] Negotiation successful
0618 22:22:22.225209 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260618 22:22:22.226661 22744 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.124" port: 35989 } has no permanent_uuid. Determining permanent_uuid...
I20260618 22:22:22.227322 22748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:22.227066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:35989 (local address 127.0.0.1:34846)
0618 22:22:22.227186 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0618 22:22:22.227250 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:35989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260618 22:22:22.227938 22748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:22.227691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:35989 (local address 127.0.0.1:34860)
0618 22:22:22.227830 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0618 22:22:22.227881 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:35989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260618 22:22:22.228068 22744 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.124:35989: Network error: Client connection negotiation failed: client connection to 127.0.98.124:35989: connect: Connection refused (error 111)
I20260618 22:22:22.259243 22744 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.124" port: 35989 } attempt: 1
I20260618 22:22:22.260152 22748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:22.259836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:35989 (local address 127.0.0.1:34876)
0618 22:22:22.260006 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0618 22:22:22.260071 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:35989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260618 22:22:22.260977 22748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:22.260665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:35989 (local address 127.0.0.1:34880)
0618 22:22:22.260856 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0618 22:22:22.260905 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:35989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
W20260618 22:22:22.261162 22744 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.124:35989: Network error: Client connection negotiation failed: client connection to 127.0.98.124:35989: connect: Connection refused (error 111)
I20260618 22:22:22.265036 22819 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.124" port: 35989 } attempt: 1
I20260618 22:22:22.265717 22822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:22.265446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:35989 (local address 127.0.0.1:34894)
0618 22:22:22.265584 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0618 22:22:22.265644 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:35989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260618 22:22:22.266453 22822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:22.266246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:35989 (local address 127.0.0.1:34906)
0618 22:22:22.266346 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0618 22:22:22.266397 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:35989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260618 22:22:22.266582 22819 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.124:35989: Network error: Client connection negotiation failed: client connection to 127.0.98.124:35989: connect: Connection refused (error 111)
W20260618 22:22:22.326624 22825 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260618 22:22:22.326876 22825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:22:22.326900 22825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:22:22.326936 22825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:22:22.330880 22825 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:22:22.330928 22825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:22:22.330947 22825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:22:22.330986 22825 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:22:22.331018 22825 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:22:22.334743 22819 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.124" port: 35989 } attempt: 2
I20260618 22:22:22.335517 22822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:22.335208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:35989 (local address 127.0.0.1:34912)
0618 22:22:22.335363 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0618 22:22:22.335431 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:35989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260618 22:22:22.336213 22822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:22.336009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:35989 (local address 127.0.0.1:34926)
0618 22:22:22.336112 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0618 22:22:22.336159 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:35989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260618 22:22:22.336352 22819 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.124:35989: Network error: Client connection negotiation failed: client connection to 127.0.98.124:35989: connect: Connection refused (error 111)
I20260618 22:22:22.336340 22744 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.124" port: 35989 } attempt: 2
I20260618 22:22:22.336494 22825 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:43985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821065380026-393-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821065380026-393-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:43549,127.0.98.125:33567,127.0.98.124:35989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821065380026-393-0/krb5kdc/kudu_127.0.98.124.keytab
--principal=kudu/127.0.98.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.124:35989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821065380026-393-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821065380026-393-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:22:22.336959 22748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:22.336739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:35989 (local address 127.0.0.1:34940)
0618 22:22:22.336840 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0618 22:22:22.336896 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:35989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260618 22:22:22.337527 22825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:22:22.337575 22748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:22.337362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:35989 (local address 127.0.0.1:34956)
0618 22:22:22.337475 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0618 22:22:22.337523 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:35989: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260618 22:22:22.337702 22744 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.124:35989: Network error: Client connection negotiation failed: client connection to 127.0.98.124:35989: connect: Connection refused (error 111)
I20260618 22:22:22.338671 22825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:22:22.346735 22831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:22:22.346750 22832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:22:22.346735 22834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:22:22.347191 22825 server_base.cc:1034] running on GCE node
I20260618 22:22:22.347656 22825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:22:22.348640 22825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:22:22.349790 22825 hybrid_clock.cc:648] HybridClock initialized: now 1781821342349764 us; error 39 us; skew 500 ppm
Jun 18 22:22:22 dist-test-slave-g6r4 krb5kdc[22663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821342, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:22:22.352385 22825 init.cc:377] Logged in from keytab as kudu/127.0.98.124@KRBTEST.COM (short username kudu)
I20260618 22:22:22.353217 22825 webserver.cc:460] Webserver started at http://127.0.98.124:33385/ using document root <none> and password file <none>
I20260618 22:22:22.353756 22825 fs_manager.cc:359] Metadata directory not provided
I20260618 22:22:22.353842 22825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:22:22.354084 22825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:22:22.356046 22825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821065380026-393-0/minicluster-data/master-2/data/instance:
uuid: "341975eff3964773b8f5f1f507558c51"
format_stamp: "Formatted at 2026-06-18 22:22:22 on dist-test-slave-g6r4"
server_key: "2adeecad050dce33ccd81137e425294f"
server_key_iv: "f076f010236aabe0e178fdb1162b87c4"
server_key_version: "encryptionkey@0"
I20260618 22:22:22.356599 22825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821065380026-393-0/minicluster-data/master-2/wal/instance:
uuid: "341975eff3964773b8f5f1f507558c51"
format_stamp: "Formatted at 2026-06-18 22:22:22 on dist-test-slave-g6r4"
server_key: "2adeecad050dce33ccd81137e425294f"
server_key_iv: "f076f010236aabe0e178fdb1162b87c4"
server_key_version: "encryptionkey@0"
I20260618 22:22:22.360488 22825 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260618 22:22:22.362841 22841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:22:22.364073 22825 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:22:22.364219 22825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821065380026-393-0/minicluster-data/master-2/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821065380026-393-0/minicluster-data/master-2/wal
uuid: "341975eff3964773b8f5f1f507558c51"
format_stamp: "Formatted at 2026-06-18 22:22:22 on dist-test-slave-g6r4"
server_key: "2adeecad050dce33ccd81137e425294f"
server_key_iv: "f076f010236aabe0e178fdb1162b87c4"
server_key_version: "encryptionkey@0"
I20260618 22:22:22.364362 22825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821065380026-393-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821065380026-393-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821065380026-393-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:22:22.376153 22825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:22:22.378800 22825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:22:22.378980 22825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:22:22.385591 22825 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.124:35989
I20260618 22:22:22.385989 22893 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.124:35989 every 8 connection(s)
I20260618 22:22:22.386548 22825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821065380026-393-0/minicluster-data/master-2/data/info.pb
I20260618 22:22:22.389284 22894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:22:22.389707 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 22825
I20260618 22:22:22.389837 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781821065380026-393-0/minicluster-data/master-2/wal/instance
I20260618 22:22:22.390125 393 external_mini_cluster.cc:1428] Setting key 00f4c6872f27e419e6f23b1dce0f0365
I20260618 22:22:22.394196 22894 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.126" port: 43549 } has no permanent_uuid. Determining permanent_uuid...
Jun 18 22:22:22 dist-test-slave-g6r4 krb5kdc[22663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821341, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.124@KRBTEST.COM
Jun 18 22:22:22 dist-test-slave-g6r4 krb5kdc[22663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821342, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.124@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:22:22.404354 22746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:22.395798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55498 (local address 127.0.98.126:43549)
0618 22:22:22.395929 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:22.395933 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:22.396270 (+ 337us) server_negotiation.cc:407] Connection header received
0618 22:22:22.396395 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:22.396398 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:22.396447 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:22.396539 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:22.397108 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:22.397582 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:22.398481 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:22.398690 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:22.398800 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:22.401058 (+ 2258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:22:22.401064 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:22:22.401068 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:22:22.401096 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:22:22.402560 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:22.403066 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:22.403070 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:22.403072 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:22.403148 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:22.403437 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:22.403441 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:22.403446 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:22.403674 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:22:22.403772 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0618 22:22:22.404098 (+ 326us) server_negotiation.cc:299] Negotiation successful
0618 22:22:22.404201 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260618 22:22:22.404860 22898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:22.395710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:43549 (local address 127.0.0.1:55498)
0618 22:22:22.396141 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0618 22:22:22.396187 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0618 22:22:22.396309 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:22:22.396572 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:22:22.396576 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:22:22.396878 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:22:22.396992 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:22:22.396997 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:22:22.397696 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:22:22.397699 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:22:22.398364 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:22:22.398373 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:22:22.398824 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:22:22.398828 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:22:22.398933 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:22.399430 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:22:22.399449 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:22:22.400917 (+ 1468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:22:22.402683 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:22:22.402691 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:22:22.402697 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:22:22.402980 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:22:22.403234 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:22:22.403236 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:22:22.403237 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:22:22.403345 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:22:22.403788 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:22:22.403795 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:22:22.403909 (+ 114us) client_negotiation.cc:769] Sending connection context
0618 22:22:22.404031 (+ 122us) client_negotiation.cc:240] Negotiation successful
0618 22:22:22.404150 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":172,"threads_started":1}
I20260618 22:22:22.404861 22897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:22.391326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34966 (local address 127.0.98.124:35989)
0618 22:22:22.391694 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:22.391708 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:22.391741 (+ 33us) server_negotiation.cc:407] Connection header received
0618 22:22:22.392512 (+ 771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:22.392530 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:22.392853 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:22.393113 (+ 260us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:22.394114 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:22.395106 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:22.396177 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:22.396817 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:22.396983 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:22.399629 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:22:22.399634 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:22:22.399642 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:22:22.399662 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:22:22.401998 (+ 2336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:22.402420 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:22.402427 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:22.402436 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:22.402565 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:22.402901 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:22.402903 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:22.402904 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:22.403349 (+ 445us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:22:22.403573 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0618 22:22:22.403759 (+ 186us) server_negotiation.cc:299] Negotiation successful
0618 22:22:22.404058 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":239,"thread_start_us":132,"threads_started":1}
I20260618 22:22:22.406073 22894 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.125" port: 33567 } has no permanent_uuid. Determining permanent_uuid...
Jun 18 22:22:22 dist-test-slave-g6r4 krb5kdc[22663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821342, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.124@KRBTEST.COM for kudu/127.0.98.125@KRBTEST.COM
Jun 18 22:22:22 dist-test-slave-g6r4 krb5kdc[22663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821341, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:22:22.414086 22824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:22.406612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44828 (local address 127.0.98.125:33567)
0618 22:22:22.406746 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:22.406750 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:22.406763 (+ 13us) server_negotiation.cc:407] Connection header received
0618 22:22:22.406814 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:22.406817 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:22.406860 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:22.406943 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:22.407370 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:22.407863 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:22.408596 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:22.408823 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:22.408937 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:22.410850 (+ 1913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:22:22.410855 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:22:22.410860 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:22:22.410895 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:22:22.412291 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:22.412744 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:22.412749 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:22.412752 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:22.412868 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:22.413181 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:22.413186 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:22.413188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:22.413421 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:22:22.413506 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0618 22:22:22.413805 (+ 299us) server_negotiation.cc:299] Negotiation successful
0618 22:22:22.413975 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260618 22:22:22.414094 22898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:22.406482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:33567 (local address 127.0.0.1:44828)
0618 22:22:22.406617 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0618 22:22:22.406633 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 22:22:22.406696 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:22:22.406981 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:22:22.406987 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:22:22.407154 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:22:22.407242 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:22:22.407248 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:22:22.407974 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:22:22.407979 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:22:22.408483 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:22:22.408490 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:22:22.408944 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:22:22.408948 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:22:22.409029 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:22.409490 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:22:22.409503 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:22:22.410706 (+ 1203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:22:22.412382 (+ 1676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:22:22.412387 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:22:22.412391 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:22:22.412637 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:22:22.412981 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:22:22.412985 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:22:22.412988 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:22:22.413086 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:22:22.413523 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:22:22.413529 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:22:22.413636 (+ 107us) client_negotiation.cc:769] Sending connection context
0618 22:22:22.413745 (+ 109us) client_negotiation.cc:240] Negotiation successful
0618 22:22:22.413864 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260618 22:22:22.414835 22894 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.124" port: 35989 } has no permanent_uuid. Determining permanent_uuid...
I20260618 22:22:22.416067 22746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:22.407156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55500 (local address 127.0.98.126:43549)
0618 22:22:22.407258 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:22.407265 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:22.407278 (+ 13us) server_negotiation.cc:407] Connection header received
0618 22:22:22.407456 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:22.407459 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:22.407496 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:22.407574 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:22.408115 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:22.408576 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:22.409386 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:22.409732 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:22.409840 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:22.412318 (+ 2478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:22:22.412322 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:22:22.412325 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:22:22.412352 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:22:22.414182 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:22.414626 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:22.414629 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:22.414630 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:22.414722 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:22.414997 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:22.415001 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:22.415003 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:22.415190 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:22:22.415432 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0618 22:22:22.415769 (+ 337us) server_negotiation.cc:299] Negotiation successful
0618 22:22:22.415889 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
Jun 18 22:22:22 dist-test-slave-g6r4 krb5kdc[22663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821342, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.124@KRBTEST.COM for kudu/127.0.98.124@KRBTEST.COM
Jun 18 22:22:22 dist-test-slave-g6r4 krb5kdc[22663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821341, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.125@KRBTEST.COM
I20260618 22:22:22.423614 22897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:22.415357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34974 (local address 127.0.98.124:35989)
0618 22:22:22.415464 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:22.415468 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:22.415480 (+ 12us) server_negotiation.cc:407] Connection header received
0618 22:22:22.415509 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:22.415512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:22.415550 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:22.415633 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:22.416011 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:22.416702 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:22.417583 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:22.418042 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:22.418153 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:22.420285 (+ 2132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:22:22.420288 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:22:22.420290 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:22:22.420310 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:22:22.421945 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:22.422421 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:22.422424 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:22.422425 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:22.422491 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:22.422757 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:22.422762 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:22.422764 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:22.422925 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:22:22.423034 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0618 22:22:22.423258 (+ 224us) server_negotiation.cc:299] Negotiation successful
0618 22:22:22.423429 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260618 22:22:22.423874 22898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:22.415247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:35989 (local address 127.0.0.1:34974)
0618 22:22:22.415359 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0618 22:22:22.415373 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:22:22.415443 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:22:22.415642 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:22:22.415644 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:22:22.415811 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:22:22.415898 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:22:22.415904 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:22:22.416798 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:22:22.416804 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:22:22.417475 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:22:22.417483 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:22:22.418177 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:22:22.418180 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:22:22.418286 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:22.418724 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:22:22.418737 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:22:22.420164 (+ 1427us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:22:22.422074 (+ 1910us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:22:22.422077 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:22:22.422080 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:22:22.422320 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:22:22.422575 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:22:22.422578 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:22:22.422580 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:22:22.422675 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:22:22.423034 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:22:22.423037 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:22:22.423112 (+ 75us) client_negotiation.cc:769] Sending connection context
0618 22:22:22.423216 (+ 104us) client_negotiation.cc:240] Negotiation successful
0618 22:22:22.423731 (+ 515us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":345}
I20260618 22:22:22.426393 22824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:22.417183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44838 (local address 127.0.98.125:33567)
0618 22:22:22.417293 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:22.417297 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:22.417336 (+ 39us) server_negotiation.cc:407] Connection header received
0618 22:22:22.417443 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:22.417446 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:22.417485 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:22.417562 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:22.418293 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:22.419025 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:22.420077 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:22.420406 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:22.420481 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:22.423146 (+ 2665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:22:22.423158 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:22:22.423160 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:22:22.423182 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:22:22.424817 (+ 1635us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:22.425222 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:22.425227 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:22.425228 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:22.425304 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:22.425589 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:22.425594 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:22.425596 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:22.425760 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:22:22.426017 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0618 22:22:22.426133 (+ 116us) server_negotiation.cc:299] Negotiation successful
0618 22:22:22.426242 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260618 22:22:22.427125 22894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 341975eff3964773b8f5f1f507558c51: Bootstrap starting.
I20260618 22:22:22.427515 22819 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.124" port: 35989 } attempt: 3
I20260618 22:22:22.429574 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:22:22.429975 22894 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 341975eff3964773b8f5f1f507558c51: Neither blocks nor log segments found. Creating new log.
I20260618 22:22:22.430768 22894 log.cc:824] T 00000000000000000000000000000000 P 341975eff3964773b8f5f1f507558c51: Log is configured to *not* fsync() on all Append() calls
I20260618 22:22:22.430915 22718 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "1116e0c1f1d046b3bac47ad94de725a6"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:55500
Jun 18 22:22:22 dist-test-slave-g6r4 krb5kdc[22663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821342, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.125@KRBTEST.COM for kudu/127.0.98.124@KRBTEST.COM
I20260618 22:22:22.432832 22894 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 341975eff3964773b8f5f1f507558c51: No bootstrap required, opened a new log
I20260618 22:22:22.436190 22822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:22.427850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:35989 (local address 127.0.0.1:34990)
0618 22:22:22.427950 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0618 22:22:22.427964 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:22:22.428031 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:22:22.428331 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:22:22.428334 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:22:22.428524 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:22:22.428615 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:22:22.428621 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:22:22.429603 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:22:22.429607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:22:22.430405 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:22:22.430417 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:22:22.431003 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:22:22.431007 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:22:22.431090 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:22.431572 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:22:22.431586 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:22:22.432968 (+ 1382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:22:22.434655 (+ 1687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:22:22.434658 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:22:22.434660 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:22:22.434901 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:22:22.435140 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:22:22.435144 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:22:22.435146 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:22:22.435234 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:22:22.435689 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:22:22.435692 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:22:22.435785 (+ 93us) client_negotiation.cc:769] Sending connection context
0618 22:22:22.435892 (+ 107us) client_negotiation.cc:240] Negotiation successful
0618 22:22:22.436027 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260618 22:22:22.436208 22897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:22.427949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34990 (local address 127.0.98.124:35989)
0618 22:22:22.428123 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:22.428127 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:22.428138 (+ 11us) server_negotiation.cc:407] Connection header received
0618 22:22:22.428170 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:22.428172 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:22.428208 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:22.428722 (+ 514us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:22.428800 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:22.429467 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:22.430541 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:22.430861 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:22.430994 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:22.433105 (+ 2111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:22:22.433109 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:22:22.433112 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:22:22.433136 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:22:22.434560 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:22.434994 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:22.434996 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:22.434997 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:22.435061 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:22.435350 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:22.435353 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:22.435355 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:22.435574 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:22:22.435669 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0618 22:22:22.435957 (+ 288us) server_negotiation.cc:299] Negotiation successful
0618 22:22:22.436060 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260618 22:22:22.436826 22744 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.124" port: 35989 } attempt: 3
I20260618 22:22:22.436997 22894 raft_consensus.cc:348] T 00000000000000000000000000000000 P 341975eff3964773b8f5f1f507558c51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1116e0c1f1d046b3bac47ad94de725a6" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 43549 } } peers { permanent_uuid: "080ee8a2d01e4028a36ab076df4fbc75" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 33567 } } peers { permanent_uuid: "341975eff3964773b8f5f1f507558c51" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 35989 } }
I20260618 22:22:22.437233 22894 raft_consensus.cc:729] T 00000000000000000000000000000000 P 341975eff3964773b8f5f1f507558c51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 341975eff3964773b8f5f1f507558c51, State: Initialized, Role: FOLLOWER
I20260618 22:22:22.437567 22894 consensus_queue.cc:260] T 00000000000000000000000000000000 P 341975eff3964773b8f5f1f507558c51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1116e0c1f1d046b3bac47ad94de725a6" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 43549 } } peers { permanent_uuid: "080ee8a2d01e4028a36ab076df4fbc75" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 33567 } } peers { permanent_uuid: "341975eff3964773b8f5f1f507558c51" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 35989 } }
I20260618 22:22:22.438153 22900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 341975eff3964773b8f5f1f507558c51 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1116e0c1f1d046b3bac47ad94de725a6" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 43549 } } peers { permanent_uuid: "080ee8a2d01e4028a36ab076df4fbc75" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 33567 } } peers { permanent_uuid: "341975eff3964773b8f5f1f507558c51" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 35989 } } }
I20260618 22:22:22.438370 22900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 341975eff3964773b8f5f1f507558c51 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:22:22.439229 22894 sys_catalog.cc:564] T 00000000000000000000000000000000 P 341975eff3964773b8f5f1f507558c51 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:22:22.439263 22819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 080ee8a2d01e4028a36ab076df4fbc75: Bootstrap starting.
Jun 18 22:22:22 dist-test-slave-g6r4 krb5kdc[22663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821341, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for kudu/127.0.98.124@KRBTEST.COM
I20260618 22:22:22.442337 22819 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 080ee8a2d01e4028a36ab076df4fbc75: Neither blocks nor log segments found. Creating new log.
I20260618 22:22:22.443140 22819 log.cc:824] T 00000000000000000000000000000000 P 080ee8a2d01e4028a36ab076df4fbc75: Log is configured to *not* fsync() on all Append() calls
W20260618 22:22:22.445412 22911 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 341975eff3964773b8f5f1f507558c51: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 22:22:22.445488 22911 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 22:22:22.447124 22748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:22.437197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:35989 (local address 127.0.0.1:35004)
0618 22:22:22.437377 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0618 22:22:22.437391 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 22:22:22.437478 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:22:22.437704 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:22:22.437707 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:22:22.437883 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:22:22.438044 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:22:22.438056 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:22:22.439020 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:22:22.439023 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:22:22.439820 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:22:22.439831 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:22:22.440464 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:22:22.440468 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:22:22.440557 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:22.441080 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:22:22.441094 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:22:22.442729 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:22:22.444978 (+ 2249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:22:22.444981 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:22:22.444983 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:22:22.445493 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:22:22.445952 (+ 459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:22:22.445955 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:22:22.445958 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:22:22.446070 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:22:22.446579 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:22:22.446582 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:22:22.446697 (+ 115us) client_negotiation.cc:769] Sending connection context
0618 22:22:22.446808 (+ 111us) client_negotiation.cc:240] Negotiation successful
0618 22:22:22.446949 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104}
I20260618 22:22:22.447296 22819 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 080ee8a2d01e4028a36ab076df4fbc75: No bootstrap required, opened a new log
I20260618 22:22:22.443810 22825 master_runner.cc:186] Error getting master registration for 127.0.98.126:43549: OK, instance_id { permanent_uuid: "1116e0c1f1d046b3bac47ad94de725a6" instance_seqno: 1781821341970553 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260618 22:22:22.447901 22897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:22.437233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35004 (local address 127.0.98.124:35989)
0618 22:22:22.437368 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:22.437372 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:22.437439 (+ 67us) server_negotiation.cc:407] Connection header received
0618 22:22:22.437551 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:22.437554 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:22.437596 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:22.437674 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:22.438182 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:22.438874 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:22.439962 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:22.440257 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:22.440352 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:22.442860 (+ 2508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:22:22.442863 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:22:22.442866 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:22:22.442890 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:22:22.444828 (+ 1938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:22.445703 (+ 875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:22.445707 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:22.445709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:22.445807 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:22.446196 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:22.446199 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:22.446201 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:22.446421 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:22:22.446515 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0618 22:22:22.447627 (+ 1112us) server_negotiation.cc:299] Negotiation successful
0618 22:22:22.447743 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260618 22:22:22.448045 22825 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260618 22:22:22.451371 22819 raft_consensus.cc:348] T 00000000000000000000000000000000 P 080ee8a2d01e4028a36ab076df4fbc75 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1116e0c1f1d046b3bac47ad94de725a6" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 43549 } } peers { permanent_uuid: "080ee8a2d01e4028a36ab076df4fbc75" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 33567 } } peers { permanent_uuid: "341975eff3964773b8f5f1f507558c51" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 35989 } }
I20260618 22:22:22.451598 22819 raft_consensus.cc:729] T 00000000000000000000000000000000 P 080ee8a2d01e4028a36ab076df4fbc75 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 080ee8a2d01e4028a36ab076df4fbc75, State: Initialized, Role: FOLLOWER
I20260618 22:22:22.451943 22819 consensus_queue.cc:260] T 00000000000000000000000000000000 P 080ee8a2d01e4028a36ab076df4fbc75 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1116e0c1f1d046b3bac47ad94de725a6" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 43549 } } peers { permanent_uuid: "080ee8a2d01e4028a36ab076df4fbc75" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 33567 } } peers { permanent_uuid: "341975eff3964773b8f5f1f507558c51" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 35989 } }
I20260618 22:22:22.452453 22914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 080ee8a2d01e4028a36ab076df4fbc75 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1116e0c1f1d046b3bac47ad94de725a6" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 43549 } } peers { permanent_uuid: "080ee8a2d01e4028a36ab076df4fbc75" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 33567 } } peers { permanent_uuid: "341975eff3964773b8f5f1f507558c51" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 35989 } } }
I20260618 22:22:22.452615 22914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 080ee8a2d01e4028a36ab076df4fbc75 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:22:22.453260 22819 sys_catalog.cc:564] T 00000000000000000000000000000000 P 080ee8a2d01e4028a36ab076df4fbc75 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:22:22.456938 22744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1116e0c1f1d046b3bac47ad94de725a6: Bootstrap starting.
I20260618 22:22:22.459815 22744 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1116e0c1f1d046b3bac47ad94de725a6: Neither blocks nor log segments found. Creating new log.
I20260618 22:22:22.460668 22744 log.cc:824] T 00000000000000000000000000000000 P 1116e0c1f1d046b3bac47ad94de725a6: Log is configured to *not* fsync() on all Append() calls
I20260618 22:22:22.460491 22751 master_runner.cc:186] Error getting master registration for 127.0.98.126:43549: OK, instance_id { permanent_uuid: "1116e0c1f1d046b3bac47ad94de725a6" instance_seqno: 1781821341970553 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260618 22:22:22.464349 22751 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260618 22:22:22.465091 22925 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 080ee8a2d01e4028a36ab076df4fbc75: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 22:22:22.465173 22925 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 22:22:22.467010 22744 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1116e0c1f1d046b3bac47ad94de725a6: No bootstrap required, opened a new log
I20260618 22:22:22.469990 22744 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1116e0c1f1d046b3bac47ad94de725a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1116e0c1f1d046b3bac47ad94de725a6" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 43549 } } peers { permanent_uuid: "080ee8a2d01e4028a36ab076df4fbc75" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 33567 } } peers { permanent_uuid: "341975eff3964773b8f5f1f507558c51" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 35989 } }
I20260618 22:22:22.470216 22744 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1116e0c1f1d046b3bac47ad94de725a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1116e0c1f1d046b3bac47ad94de725a6, State: Initialized, Role: FOLLOWER
I20260618 22:22:22.470537 22744 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1116e0c1f1d046b3bac47ad94de725a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1116e0c1f1d046b3bac47ad94de725a6" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 43549 } } peers { permanent_uuid: "080ee8a2d01e4028a36ab076df4fbc75" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 33567 } } peers { permanent_uuid: "341975eff3964773b8f5f1f507558c51" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 35989 } }
I20260618 22:22:22.471056 22928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1116e0c1f1d046b3bac47ad94de725a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1116e0c1f1d046b3bac47ad94de725a6" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 43549 } } peers { permanent_uuid: "080ee8a2d01e4028a36ab076df4fbc75" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 33567 } } peers { permanent_uuid: "341975eff3964773b8f5f1f507558c51" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 35989 } } }
I20260618 22:22:22.471331 22928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1116e0c1f1d046b3bac47ad94de725a6 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:22:22.471748 22744 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1116e0c1f1d046b3bac47ad94de725a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:22:22.471793 22718 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1116e0c1f1d046b3bac47ad94de725a6 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260618 22:22:22.471894 22718 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1116e0c1f1d046b3bac47ad94de725a6 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:22:22.473081 22718 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1116e0c1f1d046b3bac47ad94de725a6 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1116e0c1f1d046b3bac47ad94de725a6" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 43549 } } peers { permanent_uuid: "080ee8a2d01e4028a36ab076df4fbc75" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 33567 } } peers { permanent_uuid: "341975eff3964773b8f5f1f507558c51" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 35989 } }
I20260618 22:22:22.475031 22793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1116e0c1f1d046b3bac47ad94de725a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "080ee8a2d01e4028a36ab076df4fbc75"
I20260618 22:22:22.475294 22793 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 080ee8a2d01e4028a36ab076df4fbc75 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:22:22.474053 22718 leader_election.cc:288] T 00000000000000000000000000000000 P 1116e0c1f1d046b3bac47ad94de725a6 [CANDIDATE]: Term 1 election: Requested vote from peers 080ee8a2d01e4028a36ab076df4fbc75 (127.0.98.125:33567), 341975eff3964773b8f5f1f507558c51 (127.0.98.124:35989)
I20260618 22:22:22.476733 22793 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 080ee8a2d01e4028a36ab076df4fbc75 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1116e0c1f1d046b3bac47ad94de725a6 in term 1.
I20260618 22:22:22.477231 22868 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1116e0c1f1d046b3bac47ad94de725a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "341975eff3964773b8f5f1f507558c51"
W20260618 22:22:22.477545 22939 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1116e0c1f1d046b3bac47ad94de725a6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260618 22:22:22.477519 22868 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 341975eff3964773b8f5f1f507558c51 [term 0 FOLLOWER]: Advancing to term 1
W20260618 22:22:22.477988 22939 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 22:22:22.479255 22692 leader_election.cc:302] T 00000000000000000000000000000000 P 1116e0c1f1d046b3bac47ad94de725a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 080ee8a2d01e4028a36ab076df4fbc75, 1116e0c1f1d046b3bac47ad94de725a6; no voters:
I20260618 22:22:22.480226 22868 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 341975eff3964773b8f5f1f507558c51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1116e0c1f1d046b3bac47ad94de725a6 in term 1.
I20260618 22:22:22.481125 22928 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1116e0c1f1d046b3bac47ad94de725a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:22:22.481336 22928 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1116e0c1f1d046b3bac47ad94de725a6 [term 1 LEADER]: Becoming Leader. State: Replica: 1116e0c1f1d046b3bac47ad94de725a6, State: Running, Role: LEADER
I20260618 22:22:22.481645 22928 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1116e0c1f1d046b3bac47ad94de725a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1116e0c1f1d046b3bac47ad94de725a6" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 43549 } } peers { permanent_uuid: "080ee8a2d01e4028a36ab076df4fbc75" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 33567 } } peers { permanent_uuid: "341975eff3964773b8f5f1f507558c51" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 35989 } }
I20260618 22:22:22.483597 22676 master_runner.cc:263] No leader master found from master 1116e0c1f1d046b3bac47ad94de725a6
I20260618 22:22:22.483662 22676 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260618 22:22:22.484584 22928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1116e0c1f1d046b3bac47ad94de725a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1116e0c1f1d046b3bac47ad94de725a6. Latest consensus state: current_term: 1 leader_uuid: "1116e0c1f1d046b3bac47ad94de725a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1116e0c1f1d046b3bac47ad94de725a6" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 43549 } } peers { permanent_uuid: "080ee8a2d01e4028a36ab076df4fbc75" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 33567 } } peers { permanent_uuid: "341975eff3964773b8f5f1f507558c51" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 35989 } } }
I20260618 22:22:22.484694 22928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1116e0c1f1d046b3bac47ad94de725a6 [sys.catalog]: This master's current role is: LEADER
I20260618 22:22:22.491101 22947 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:22:22.492076 22947 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:22:22.497332 22868 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 341975eff3964773b8f5f1f507558c51 [term 1 FOLLOWER]: Refusing update from remote peer 1116e0c1f1d046b3bac47ad94de725a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:22:22.498034 22945 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1116e0c1f1d046b3bac47ad94de725a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "341975eff3964773b8f5f1f507558c51" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 35989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:22:22.501760 22900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 341975eff3964773b8f5f1f507558c51 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1116e0c1f1d046b3bac47ad94de725a6. Latest consensus state: current_term: 1 leader_uuid: "1116e0c1f1d046b3bac47ad94de725a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1116e0c1f1d046b3bac47ad94de725a6" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 43549 } } peers { permanent_uuid: "080ee8a2d01e4028a36ab076df4fbc75" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 33567 } } peers { permanent_uuid: "341975eff3964773b8f5f1f507558c51" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 35989 } } }
I20260618 22:22:22.501880 22900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 341975eff3964773b8f5f1f507558c51 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:22:22.502925 22793 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 080ee8a2d01e4028a36ab076df4fbc75 [term 1 FOLLOWER]: Refusing update from remote peer 1116e0c1f1d046b3bac47ad94de725a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:22:22.503520 22945 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1116e0c1f1d046b3bac47ad94de725a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "080ee8a2d01e4028a36ab076df4fbc75" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 33567 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:22:22.507063 22914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 080ee8a2d01e4028a36ab076df4fbc75 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1116e0c1f1d046b3bac47ad94de725a6. Latest consensus state: current_term: 1 leader_uuid: "1116e0c1f1d046b3bac47ad94de725a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1116e0c1f1d046b3bac47ad94de725a6" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 43549 } } peers { permanent_uuid: "080ee8a2d01e4028a36ab076df4fbc75" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 33567 } } peers { permanent_uuid: "341975eff3964773b8f5f1f507558c51" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 35989 } } }
I20260618 22:22:22.507189 22914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 080ee8a2d01e4028a36ab076df4fbc75 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:22:22.508550 22824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:22.490274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44850 (local address 127.0.98.125:33567)
0618 22:22:22.490385 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:22.490389 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:22.497691 (+ 7302us) server_negotiation.cc:407] Connection header received
0618 22:22:22.497877 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:22.497880 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:22.497953 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:22.498004 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:22.499079 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:22.499872 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:22.501045 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:22.501382 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:22.501449 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:22.503447 (+ 1998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:22:22.503451 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:22:22.503454 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:22:22.503478 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:22:22.505169 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:22.505639 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:22.505643 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:22.505645 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:22.505720 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:22.507328 (+ 1608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:22.507332 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:22.507334 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:22.507538 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:22:22.507634 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0618 22:22:22.508296 (+ 662us) server_negotiation.cc:299] Negotiation successful
0618 22:22:22.508404 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260618 22:22:22.513592 22945 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1116e0c1f1d046b3bac47ad94de725a6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1116e0c1f1d046b3bac47ad94de725a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1116e0c1f1d046b3bac47ad94de725a6" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 43549 } } peers { permanent_uuid: "080ee8a2d01e4028a36ab076df4fbc75" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 33567 } } peers { permanent_uuid: "341975eff3964773b8f5f1f507558c51" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 35989 } } }
I20260618 22:22:22.513725 22945 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1116e0c1f1d046b3bac47ad94de725a6 [sys.catalog]: This master's current role is: LEADER
I20260618 22:22:22.514751 22914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 080ee8a2d01e4028a36ab076df4fbc75 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1116e0c1f1d046b3bac47ad94de725a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1116e0c1f1d046b3bac47ad94de725a6" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 43549 } } peers { permanent_uuid: "080ee8a2d01e4028a36ab076df4fbc75" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 33567 } } peers { permanent_uuid: "341975eff3964773b8f5f1f507558c51" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 35989 } } }
I20260618 22:22:22.514861 22914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 080ee8a2d01e4028a36ab076df4fbc75 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:22:22.515383 22951 mvcc.cc:205] Tried to move back new op lower bound from 7298340218861752320 to 7298340218813063168. Current Snapshot: MvccSnapshot[applied={T|T < 7298340218861752320}]
I20260618 22:22:22.517982 22928 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1116e0c1f1d046b3bac47ad94de725a6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1116e0c1f1d046b3bac47ad94de725a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1116e0c1f1d046b3bac47ad94de725a6" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 43549 } } peers { permanent_uuid: "080ee8a2d01e4028a36ab076df4fbc75" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 33567 } } peers { permanent_uuid: "341975eff3964773b8f5f1f507558c51" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 35989 } } }
I20260618 22:22:22.518095 22928 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1116e0c1f1d046b3bac47ad94de725a6 [sys.catalog]: This master's current role is: LEADER
I20260618 22:22:22.519240 22900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 341975eff3964773b8f5f1f507558c51 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1116e0c1f1d046b3bac47ad94de725a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1116e0c1f1d046b3bac47ad94de725a6" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 43549 } } peers { permanent_uuid: "080ee8a2d01e4028a36ab076df4fbc75" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 33567 } } peers { permanent_uuid: "341975eff3964773b8f5f1f507558c51" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 35989 } } }
I20260618 22:22:22.519379 22900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 341975eff3964773b8f5f1f507558c51 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:22:22.519939 22954 mvcc.cc:205] Tried to move back new op lower bound from 7298340218861752320 to 7298340218813063168. Current Snapshot: MvccSnapshot[applied={T|T < 7298340218861752320}]
I20260618 22:22:22.522996 22952 mvcc.cc:205] Tried to move back new op lower bound from 7298340218861752320 to 7298340218813063168. Current Snapshot: MvccSnapshot[applied={T|T < 7298340218861752320}]
I20260618 22:22:22.525636 22947 catalog_manager.cc:1347] Generated new cluster ID: 0357ee925c434ad19a0cf6ffaeb1f46f
I20260618 22:22:22.525707 22947 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:22:22.527238 22746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:22.490399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55516 (local address 127.0.98.126:43549)
0618 22:22:22.490512 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:22.490517 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:22.502152 (+ 11635us) server_negotiation.cc:407] Connection header received
0618 22:22:22.502362 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:22.502365 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:22.502415 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:22.502469 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:22.504783 (+ 2314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:22.505558 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:22.508807 (+ 3249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:22.509157 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:22.509254 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:22.520471 (+ 11217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:22:22.520475 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:22:22.520478 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:22:22.520505 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:22:22.522361 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:22.524959 (+ 2598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:22.524963 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:22.524965 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:22.525056 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:22.525777 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:22.525781 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:22.525783 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:22.526022 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:22:22.526155 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0618 22:22:22.526941 (+ 786us) server_negotiation.cc:299] Negotiation successful
0618 22:22:22.527058 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260618 22:22:22.528561 22897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:22.490910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35014 (local address 127.0.98.124:35989)
0618 22:22:22.491013 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:22.491017 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:22.503583 (+ 12566us) server_negotiation.cc:407] Connection header received
0618 22:22:22.503769 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:22.503772 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:22.503825 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:22.503884 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:22.506781 (+ 2897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:22.507571 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:22.517830 (+ 10259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:22.518239 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:22.518323 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:22.522923 (+ 4600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:22:22.522931 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:22:22.522937 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:22:22.522965 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:22:22.524806 (+ 1841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:22.526500 (+ 1694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:22.526504 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:22.526506 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:22.526594 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:22.527668 (+ 1074us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:22.527672 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:22.527674 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:22.527880 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:22:22.527980 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0618 22:22:22.528311 (+ 331us) server_negotiation.cc:299] Negotiation successful
0618 22:22:22.528420 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260618 22:22:22.561282 22947 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:22:22.562647 22947 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:22:22.581380 22947 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1116e0c1f1d046b3bac47ad94de725a6: Generated new TSK 0
I20260618 22:22:22.582290 22947 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260618 22:22:22.616205 22746 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:22:22.612837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55522 (local address 127.0.98.126:43549)
0618 22:22:22.612980 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:22.612985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:22.614760 (+ 1775us) server_negotiation.cc:407] Connection header received
0618 22:22:22.615259 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:22.615264 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:22.615320 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:22.615380 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:22.616064 (+ 684us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55522: BlockingRecv error: recv got EOF from 127.0.0.1:55522 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
W20260618 22:22:22.617733 22824 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:22:22.613739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44862 (local address 127.0.98.125:33567)
0618 22:22:22.613863 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:22.613867 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:22.616739 (+ 2872us) server_negotiation.cc:407] Connection header received
0618 22:22:22.616921 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:22.616924 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:22.616977 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:22.617037 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:22.617588 (+ 551us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44862: BlockingRecv error: recv got EOF from 127.0.0.1:44862 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260618 22:22:22.619195 22897 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:22:22.614443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35018 (local address 127.0.98.124:35989)
0618 22:22:22.614566 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:22.614570 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:22.618164 (+ 3594us) server_negotiation.cc:407] Connection header received
0618 22:22:22.618463 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:22.618467 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:22.618519 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:22.618574 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:22.619083 (+ 509us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35018: BlockingRecv error: recv got EOF from 127.0.0.1:35018 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260618 22:22:22.624182 22824 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:22:22.622434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44874 (local address 127.0.98.125:33567)
0618 22:22:22.622552 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:22.622557 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:22.623173 (+ 616us) server_negotiation.cc:407] Connection header received
0618 22:22:22.623402 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:22.623406 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:22.623477 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:22.623536 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:22.624073 (+ 537us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44874: BlockingRecv error: recv got EOF from 127.0.0.1:44874 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260618 22:22:22.629595 22824 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:22:22.627940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44880 (local address 127.0.98.125:33567)
0618 22:22:22.628046 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:22.628051 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:22.628554 (+ 503us) server_negotiation.cc:407] Connection header received
0618 22:22:22.628759 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:22.628762 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:22.628815 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:22.628871 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:22.629458 (+ 587us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44880: BlockingRecv error: recv got EOF from 127.0.0.1:44880 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260618 22:22:22.639253 22824 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:22:22.633851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44882 (local address 127.0.98.125:33567)
0618 22:22:22.633970 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:22.633974 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:22.638201 (+ 4227us) server_negotiation.cc:407] Connection header received
0618 22:22:22.638389 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:22.638393 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:22.638444 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:22.638493 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:22.639119 (+ 626us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44882: BlockingRecv error: recv got EOF from 127.0.0.1:44882 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260618 22:22:22.647328 22824 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:22:22.645516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44896 (local address 127.0.98.125:33567)
0618 22:22:22.645841 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:22.645845 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:22.646378 (+ 533us) server_negotiation.cc:407] Connection header received
0618 22:22:22.646585 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:22.646588 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:22.646640 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:22.646697 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:22.647224 (+ 527us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44896: BlockingRecv error: recv got EOF from 127.0.0.1:44896 (error 108)
Metrics: {"server-negotiator.queue_time_us":251}
W20260618 22:22:22.659374 22824 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:22:22.657669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44910 (local address 127.0.98.125:33567)
0618 22:22:22.657786 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:22.657790 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:22.658319 (+ 529us) server_negotiation.cc:407] Connection header received
0618 22:22:22.658517 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:22.658521 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:22.658596 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:22.658668 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:22.659259 (+ 591us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44910: BlockingRecv error: recv got EOF from 127.0.0.1:44910 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260618 22:22:22.679316 22824 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:22:22.677852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44924 (local address 127.0.98.125:33567)
0618 22:22:22.678025 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:22.678029 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:22.678263 (+ 234us) server_negotiation.cc:407] Connection header received
0618 22:22:22.678406 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:22.678410 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:22.678465 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:22.678564 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:22.679193 (+ 629us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44924: BlockingRecv error: recv got EOF from 127.0.0.1:44924 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
W20260618 22:22:22.714574 22824 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:22:22.713342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44938 (local address 127.0.98.125:33567)
0618 22:22:22.713515 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:22.713520 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:22.713634 (+ 114us) server_negotiation.cc:407] Connection header received
0618 22:22:22.713807 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:22.713811 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:22.713865 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:22.714000 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:22.714454 (+ 454us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44938: BlockingRecv error: recv got EOF from 127.0.0.1:44938 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
W20260618 22:22:22.781934 22824 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:22:22.780685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44944 (local address 127.0.98.125:33567)
0618 22:22:22.780858 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:22.780863 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:22.781025 (+ 162us) server_negotiation.cc:407] Connection header received
0618 22:22:22.781192 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:22.781196 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:22.781249 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:22.781335 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:22.781788 (+ 453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44944: BlockingRecv error: recv got EOF from 127.0.0.1:44944 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260618 22:22:22.912954 22824 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:22:22.911803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44954 (local address 127.0.98.125:33567)
0618 22:22:22.911969 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:22.911975 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:22.912092 (+ 117us) server_negotiation.cc:407] Connection header received
0618 22:22:22.912216 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:22.912219 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:22.912311 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:22.912396 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:22.912804 (+ 408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44954: BlockingRecv error: recv got EOF from 127.0.0.1:44954 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260618 22:22:23.172039 22824 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:22:23.170897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44960 (local address 127.0.98.125:33567)
0618 22:22:23.171046 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:23.171049 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:23.171100 (+ 51us) server_negotiation.cc:407] Connection header received
0618 22:22:23.171263 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:23.171266 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:23.171346 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:23.171425 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:23.171916 (+ 491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44960: BlockingRecv error: recv got EOF from 127.0.0.1:44960 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260618 22:22:23.446771 22911 catalog_manager.cc:1259] Loaded cluster ID: 0357ee925c434ad19a0cf6ffaeb1f46f
I20260618 22:22:23.446854 22911 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 341975eff3964773b8f5f1f507558c51: loading cluster ID for follower catalog manager: success
I20260618 22:22:23.448786 22911 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 341975eff3964773b8f5f1f507558c51: acquiring CA information for follower catalog manager: success
I20260618 22:22:23.449985 22911 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 341975eff3964773b8f5f1f507558c51: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260618 22:22:23.466269 22925 catalog_manager.cc:1259] Loaded cluster ID: 0357ee925c434ad19a0cf6ffaeb1f46f
I20260618 22:22:23.466329 22925 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 080ee8a2d01e4028a36ab076df4fbc75: loading cluster ID for follower catalog manager: success
I20260618 22:22:23.467911 22925 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 080ee8a2d01e4028a36ab076df4fbc75: acquiring CA information for follower catalog manager: success
I20260618 22:22:23.469056 22925 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 080ee8a2d01e4028a36ab076df4fbc75: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260618 22:22:23.689698 23023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:23.685993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55710 (local address 127.0.98.125:33567)
0618 22:22:23.686269 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:23.686272 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:23.686287 (+ 15us) server_negotiation.cc:407] Connection header received
0618 22:22:23.686423 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:23.686425 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:23.686543 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:23.686640 (+ 97us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:22:23.686854 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:23.687427 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:23.688273 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:23.688496 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:23.688576 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:23.688838 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:22:23.689303 (+ 465us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:22:23.689397 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0618 22:22:23.689452 (+ 55us) server_negotiation.cc:299] Negotiation successful
0618 22:22:23.689498 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":198,"thread_start_us":96,"threads_started":1}
I20260618 22:22:23.696274 23029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:23.692778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42164 (local address 127.0.98.124:35989)
0618 22:22:23.693019 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:23.693022 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:23.693035 (+ 13us) server_negotiation.cc:407] Connection header received
0618 22:22:23.693105 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:23.693107 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:23.693208 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:23.693316 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:22:23.693508 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:23.694058 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:23.694848 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:23.695088 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:23.695261 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:23.695524 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:22:23.695914 (+ 390us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:22:23.696015 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0618 22:22:23.696064 (+ 49us) server_negotiation.cc:299] Negotiation successful
0618 22:22:23.696111 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":110,"threads_started":1}
I20260618 22:22:23.702831 23023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:23.699106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55726 (local address 127.0.98.125:33567)
0618 22:22:23.699216 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:23.699220 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:23.699412 (+ 192us) server_negotiation.cc:407] Connection header received
0618 22:22:23.699597 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:23.699600 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:23.699653 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:23.699706 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:22:23.700043 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:23.700527 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:23.701633 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:23.701992 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:23.702092 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:23.702357 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:22:23.702434 (+ 77us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:22:23.702497 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0618 22:22:23.702667 (+ 170us) server_negotiation.cc:299] Negotiation successful
0618 22:22:23.702717 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260618 22:22:23.703478 23029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:23.699607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42168 (local address 127.0.98.124:35989)
0618 22:22:23.699721 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:23.699727 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:23.699890 (+ 163us) server_negotiation.cc:407] Connection header received
0618 22:22:23.700077 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:23.700080 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:23.700126 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:23.700169 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:22:23.700505 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:23.701107 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:23.702202 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:23.702526 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:23.702629 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:23.702838 (+ 209us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:22:23.702920 (+ 82us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:22:23.702993 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0618 22:22:23.703330 (+ 337us) server_negotiation.cc:299] Negotiation successful
0618 22:22:23.703388 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260618 22:22:23.703553 23036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:23.699079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33540 (local address 127.0.98.126:43549)
0618 22:22:23.699371 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:23.699375 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:23.699404 (+ 29us) server_negotiation.cc:407] Connection header received
0618 22:22:23.699495 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:23.699501 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:23.699656 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:23.699787 (+ 131us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:22:23.700016 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:23.700815 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:23.701838 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:23.702170 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:23.702281 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:23.702583 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:22:23.702888 (+ 305us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:22:23.703076 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0618 22:22:23.703355 (+ 279us) server_negotiation.cc:299] Negotiation successful
0618 22:22:23.703424 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":85,"threads_started":1}
I20260618 22:22:23.711431 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 22676
I20260618 22:22:23.717593 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 22751
I20260618 22:22:23.723378 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 22825
2026-06-18T22:22:23Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3126 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821065380026-393-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 22:22:23 dist-test-slave-g6r4 krb5kdc[23040](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 22:22:23 dist-test-slave-g6r4 krb5kdc[23040](info): set up 2 sockets
Jun 18 22:22:23 dist-test-slave-g6r4 krb5kdc[23040](info): commencing operation
krb5kdc: starting...
W20260618 22:22:24.766505 393 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821065380026-393-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821065380026-393-0/krb5kdc/test-user.keytab.
Jun 18 22:22:24 dist-test-slave-g6r4 krb5kdc[23040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T22:22:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T22:22:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab.
Entry for principal kudu/127.0.98.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab.
I20260618 22:22:24.885516 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821065380026-393-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821065380026-393-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821065380026-393-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:36343
--webserver_interface=127.0.98.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:34565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:36343,127.0.98.125:35773,127.0.98.124:36847
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 22:22:24.994751 23053 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260618 22:22:24.995079 23053 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:22:24.995147 23053 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:22:24.995226 23053 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:22:24.999229 23053 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:22:24.999312 23053 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:22:24.999357 23053 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:22:24.999395 23053 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:22:24.999432 23053 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:22:25.004565 23053 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:34565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821065380026-393-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821065380026-393-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:36343,127.0.98.125:35773,127.0.98.124:36847
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821065380026-393-0/krb5kdc/kudu_127.0.98.126.keytab
--principal=kudu/127.0.98.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.126:36343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23053
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821065380026-393-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:22:25.005609 23053 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:22:25.006608 23053 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 22:22:25.014688 23061 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:22:25.014724 23059 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:22:25.014693 23058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:22:25.015082 23053 server_base.cc:1034] running on GCE node
I20260618 22:22:25.015695 23053 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:22:25.016675 23053 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:22:25.017827 23053 hybrid_clock.cc:648] HybridClock initialized: now 1781821345017815 us; error 23 us; skew 500 ppm
Jun 18 22:22:25 dist-test-slave-g6r4 krb5kdc[23040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821345, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:22:25.020627 23053 init.cc:377] Logged in from keytab as kudu/127.0.98.126@KRBTEST.COM (short username kudu)
I20260618 22:22:25.021512 23053 webserver.cc:460] Webserver started at http://127.0.98.126:38817/ using document root <none> and password file <none>
I20260618 22:22:25.022104 23053 fs_manager.cc:359] Metadata directory not provided
I20260618 22:22:25.022166 23053 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:22:25.022379 23053 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:22:25.024348 23053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821065380026-393-0/minicluster-data/master-0/data/instance:
uuid: "38ea137015de482794155db583074c26"
format_stamp: "Formatted at 2026-06-18 22:22:25 on dist-test-slave-g6r4"
server_key: "a78941f1e1a29e6af06c035e7c0e1310"
server_key_iv: "55912894489af196a39279bade80b2b3"
server_key_version: "encryptionkey@0"
I20260618 22:22:25.024901 23053 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821065380026-393-0/minicluster-data/master-0/wal/instance:
uuid: "38ea137015de482794155db583074c26"
format_stamp: "Formatted at 2026-06-18 22:22:25 on dist-test-slave-g6r4"
server_key: "a78941f1e1a29e6af06c035e7c0e1310"
server_key_iv: "55912894489af196a39279bade80b2b3"
server_key_version: "encryptionkey@0"
I20260618 22:22:25.028453 23053 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260618 22:22:25.030776 23068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:22:25.031952 23053 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260618 22:22:25.032081 23053 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821065380026-393-0/minicluster-data/master-0/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821065380026-393-0/minicluster-data/master-0/wal
uuid: "38ea137015de482794155db583074c26"
format_stamp: "Formatted at 2026-06-18 22:22:25 on dist-test-slave-g6r4"
server_key: "a78941f1e1a29e6af06c035e7c0e1310"
server_key_iv: "55912894489af196a39279bade80b2b3"
server_key_version: "encryptionkey@0"
I20260618 22:22:25.032184 23053 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821065380026-393-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821065380026-393-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821065380026-393-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:22:25.048169 23053 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:22:25.051337 23053 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:22:25.051527 23053 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:22:25.058473 23053 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.126:36343
I20260618 22:22:25.058765 23120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.126:36343 every 8 connection(s)
I20260618 22:22:25.059507 23053 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821065380026-393-0/minicluster-data/master-0/data/info.pb
I20260618 22:22:25.061270 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 23053
I20260618 22:22:25.061399 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821065380026-393-0/minicluster-data/master-0/wal/instance
I20260618 22:22:25.061685 393 external_mini_cluster.cc:1428] Setting key 8da36bdbcb88b440da4629745624393a
I20260618 22:22:25.062180 23121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:22:25.065631 23121 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36343 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.125@KRBTEST.COM" created.
Jun 18 22:22:25 dist-test-slave-g6r4 krb5kdc[23040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821345, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:22:25.079769 23124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:25.067761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48390 (local address 127.0.98.126:36343)
0618 22:22:25.068135 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:25.068153 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:25.068407 (+ 254us) server_negotiation.cc:407] Connection header received
0618 22:22:25.069398 (+ 991us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:25.069402 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:25.069578 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:25.069755 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:25.070552 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:25.071396 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:25.072410 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:25.072754 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:25.072957 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:25.075275 (+ 2318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:22:25.075281 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:22:25.075289 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:22:25.075307 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:22:25.076854 (+ 1547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:25.077308 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:25.077313 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:25.077315 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:25.077413 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:25.077708 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:25.077711 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:25.077712 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:25.078090 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:22:25.078215 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0618 22:22:25.078549 (+ 334us) server_negotiation.cc:299] Negotiation successful
0618 22:22:25.078849 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":185,"server-negotiator.queue_time_us":241,"thread_start_us":112,"threads_started":1}
I20260618 22:22:25.079769 23125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:25.067761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:36343 (local address 127.0.0.1:48390)
0618 22:22:25.068274 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0618 22:22:25.068307 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0618 22:22:25.069093 (+ 786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:22:25.069752 (+ 659us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:22:25.069757 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:22:25.070193 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:22:25.070412 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:22:25.070424 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:22:25.071517 (+ 1093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:22:25.071522 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:22:25.072295 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:22:25.072304 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:22:25.072956 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:22:25.072963 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:22:25.073114 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:25.073689 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:22:25.073704 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:22:25.075109 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:22:25.076972 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:22:25.076981 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:22:25.076990 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:22:25.077207 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:22:25.077506 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:22:25.077508 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:22:25.077510 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:22:25.077612 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:22:25.078206 (+ 594us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:22:25.078211 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:22:25.078339 (+ 128us) client_negotiation.cc:769] Sending connection context
0618 22:22:25.078493 (+ 154us) client_negotiation.cc:240] Negotiation successful
0618 22:22:25.078802 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"mutex_wait_us":8,"spinlock_wait_cycles":8960,"thread_start_us":156,"threads_started":1}
I20260618 22:22:25.082136 23121 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.125" port: 35773 } has no permanent_uuid. Determining permanent_uuid...
I20260618 22:22:25.082844 23125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:25.082604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:35773 (local address 127.0.0.1:60328)
0618 22:22:25.082716 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0618 22:22:25.082782 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.125:35773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260618 22:22:25.083060 23069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.125:35773: connect: Connection refused (error 111)
I20260618 22:22:25.083832 23125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:25.083653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:35773 (local address 127.0.0.1:60338)
0618 22:22:25.083741 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0618 22:22:25.083783 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.125:35773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260618 22:22:25.084018 23121 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.125:35773: Network error: Client connection negotiation failed: client connection to 127.0.98.125:35773: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821065380026-393-0/krb5kdc/kudu_127.0.98.125.keytab.
Entry for principal kudu/127.0.98.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821065380026-393-0/krb5kdc/kudu_127.0.98.125.keytab.
I20260618 22:22:25.086900 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821065380026-393-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821065380026-393-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821065380026-393-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821065380026-393-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.125:35773
--webserver_interface=127.0.98.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821065380026-393-0/krb5kdc/kudu_127.0.98.125.keytab
--principal=kudu/127.0.98.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:34565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:36343,127.0.98.125:35773,127.0.98.124:36847
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 22:22:25.133244 23121 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.125" port: 35773 } attempt: 1
I20260618 22:22:25.134187 23125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:25.133827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:35773 (local address 127.0.0.1:60344)
0618 22:22:25.134012 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0618 22:22:25.134085 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.125:35773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260618 22:22:25.134994 23125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:25.134765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:35773 (local address 127.0.0.1:60352)
0618 22:22:25.134869 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0618 22:22:25.134923 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.125:35773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260618 22:22:25.135139 23121 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.125:35773: Network error: Client connection negotiation failed: client connection to 127.0.98.125:35773: connect: Connection refused (error 111)
W20260618 22:22:25.191614 23128 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260618 22:22:25.191866 23128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:22:25.191888 23128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:22:25.191924 23128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:22:25.195791 23128 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:22:25.195839 23128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:22:25.195863 23128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:22:25.195879 23128 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:22:25.195894 23128 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:22:25.200778 23128 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:34565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821065380026-393-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821065380026-393-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:36343,127.0.98.125:35773,127.0.98.124:36847
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821065380026-393-0/krb5kdc/kudu_127.0.98.125.keytab
--principal=kudu/127.0.98.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.125:35773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821065380026-393-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23128
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821065380026-393-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:22:25.201685 23128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:22:25.202651 23128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260618 22:22:25.203325 23121 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.125" port: 35773 } attempt: 2
I20260618 22:22:25.204113 23125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:25.203805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:35773 (local address 127.0.0.1:60366)
0618 22:22:25.203964 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0618 22:22:25.204030 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.125:35773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260618 22:22:25.204813 23125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:25.204600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:35773 (local address 127.0.0.1:60370)
0618 22:22:25.204701 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0618 22:22:25.204752 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.125:35773: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260618 22:22:25.204931 23121 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.125:35773: Network error: Client connection negotiation failed: client connection to 127.0.98.125:35773: connect: Connection refused (error 111)
W20260618 22:22:25.210989 23134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:22:25.210978 23133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:22:25.211297 23128 server_base.cc:1034] running on GCE node
W20260618 22:22:25.211440 23136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:22:25.211931 23128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:22:25.212934 23128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:22:25.214107 23128 hybrid_clock.cc:648] HybridClock initialized: now 1781821345214093 us; error 25 us; skew 500 ppm
Jun 18 22:22:25 dist-test-slave-g6r4 krb5kdc[23040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821345, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:22:25.216862 23128 init.cc:377] Logged in from keytab as kudu/127.0.98.125@KRBTEST.COM (short username kudu)
I20260618 22:22:25.217657 23128 webserver.cc:460] Webserver started at http://127.0.98.125:42007/ using document root <none> and password file <none>
I20260618 22:22:25.218242 23128 fs_manager.cc:359] Metadata directory not provided
I20260618 22:22:25.218304 23128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:22:25.218501 23128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:22:25.220438 23128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821065380026-393-0/minicluster-data/master-1/data/instance:
uuid: "d0e004157c4e47c8af9a41fba626f831"
format_stamp: "Formatted at 2026-06-18 22:22:25 on dist-test-slave-g6r4"
server_key: "0662d9ad21deaf3739a5f97689d520cd"
server_key_iv: "4088357ee314c4f6e7b1e85ccb63ed6a"
server_key_version: "encryptionkey@0"
I20260618 22:22:25.221000 23128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821065380026-393-0/minicluster-data/master-1/wal/instance:
uuid: "d0e004157c4e47c8af9a41fba626f831"
format_stamp: "Formatted at 2026-06-18 22:22:25 on dist-test-slave-g6r4"
server_key: "0662d9ad21deaf3739a5f97689d520cd"
server_key_iv: "4088357ee314c4f6e7b1e85ccb63ed6a"
server_key_version: "encryptionkey@0"
I20260618 22:22:25.224599 23128 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260618 22:22:25.227087 23143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:22:25.228312 23128 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260618 22:22:25.228431 23128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821065380026-393-0/minicluster-data/master-1/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821065380026-393-0/minicluster-data/master-1/wal
uuid: "d0e004157c4e47c8af9a41fba626f831"
format_stamp: "Formatted at 2026-06-18 22:22:25 on dist-test-slave-g6r4"
server_key: "0662d9ad21deaf3739a5f97689d520cd"
server_key_iv: "4088357ee314c4f6e7b1e85ccb63ed6a"
server_key_version: "encryptionkey@0"
I20260618 22:22:25.228533 23128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821065380026-393-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821065380026-393-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821065380026-393-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:22:25.250046 23128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:22:25.253091 23128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:22:25.253279 23128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:22:25.260152 23128 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.125:35773
I20260618 22:22:25.260471 23195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.125:35773 every 8 connection(s)
I20260618 22:22:25.261106 23128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821065380026-393-0/minicluster-data/master-1/data/info.pb
I20260618 22:22:25.262159 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 23128
I20260618 22:22:25.262302 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821065380026-393-0/minicluster-data/master-1/wal/instance
I20260618 22:22:25.262569 393 external_mini_cluster.cc:1428] Setting key 2c48f3870bf4851d138fd35ca3ff0ae7
I20260618 22:22:25.264151 23196 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:22:25.268043 23196 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36343 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.124@KRBTEST.COM" created.
Jun 18 22:22:25 dist-test-slave-g6r4 krb5kdc[23040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821345, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.125@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:22:25.282837 23124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:25.270063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48392 (local address 127.0.98.126:36343)
0618 22:22:25.270235 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:25.270239 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:25.271072 (+ 833us) server_negotiation.cc:407] Connection header received
0618 22:22:25.272111 (+ 1039us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:25.272115 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:25.272173 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:25.272279 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:25.273810 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:25.274432 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:25.275803 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:25.276247 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:25.276383 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:25.278801 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:22:25.278805 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:22:25.278808 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:22:25.278842 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:22:25.280538 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:25.281049 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:25.281052 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:25.281054 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:25.281176 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:25.281557 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:25.281560 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:25.281561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:25.281736 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:22:25.281860 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0618 22:22:25.282482 (+ 622us) server_negotiation.cc:299] Negotiation successful
0618 22:22:25.282628 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260618 22:22:25.283672 23199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:25.270421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:36343 (local address 127.0.0.1:48392)
0618 22:22:25.270927 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0618 22:22:25.270970 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0618 22:22:25.271909 (+ 939us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:22:25.272445 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:22:25.272451 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:22:25.272948 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:22:25.273644 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:22:25.273664 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:22:25.274573 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:22:25.274578 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:22:25.275676 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:22:25.275685 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:22:25.276428 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:22:25.276432 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:22:25.276570 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:25.277180 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:22:25.277196 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:22:25.278627 (+ 1431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:22:25.280655 (+ 2028us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:22:25.280660 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:22:25.280676 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:22:25.280938 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:22:25.281309 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:22:25.281312 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:22:25.281313 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:22:25.281465 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:22:25.281868 (+ 403us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:22:25.281879 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:22:25.282202 (+ 323us) client_negotiation.cc:769] Sending connection context
0618 22:22:25.282378 (+ 176us) client_negotiation.cc:240] Negotiation successful
0618 22:22:25.282681 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":156,"threads_started":1}
I20260618 22:22:25.285144 23196 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.125" port: 35773 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821065380026-393-0/krb5kdc/kudu_127.0.98.124.keytab.
Entry for principal kudu/127.0.98.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821065380026-393-0/krb5kdc/kudu_127.0.98.124.keytab.
I20260618 22:22:25.289098 393 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
/tmp/dist-test-taskIMAyAx/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821065380026-393-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821065380026-393-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821065380026-393-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821065380026-393-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.124:36847
--webserver_interface=127.0.98.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821065380026-393-0/krb5kdc/kudu_127.0.98.124.keytab
--principal=kudu/127.0.98.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.84:34565
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.126:36343,127.0.98.125:35773,127.0.98.124:36847
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821065380026-393-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821065380026-393-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821065380026-393-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 18 22:22:25 dist-test-slave-g6r4 krb5kdc[23040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821345, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.125@KRBTEST.COM for kudu/127.0.98.125@KRBTEST.COM
I20260618 22:22:25.295595 23199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:25.285620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:35773 (local address 127.0.0.1:60380)
0618 22:22:25.285730 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0618 22:22:25.285745 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 22:22:25.285819 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:22:25.286509 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:22:25.286511 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:22:25.286676 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:22:25.286770 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:22:25.286774 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:22:25.287871 (+ 1097us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:22:25.287874 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:22:25.288530 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:22:25.288544 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:22:25.289178 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:22:25.289184 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:22:25.289320 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:25.289813 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:22:25.289833 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:22:25.291613 (+ 1780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:22:25.294005 (+ 2392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:22:25.294008 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:22:25.294009 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:22:25.294216 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:22:25.294544 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:22:25.294546 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:22:25.294547 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:22:25.294624 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:22:25.295193 (+ 569us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:22:25.295196 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:22:25.295279 (+ 83us) client_negotiation.cc:769] Sending connection context
0618 22:22:25.295366 (+ 87us) client_negotiation.cc:240] Negotiation successful
0618 22:22:25.295468 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260618 22:22:25.295922 23201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:25.285779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60380 (local address 127.0.98.125:35773)
0618 22:22:25.286110 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:25.286119 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:25.286139 (+ 20us) server_negotiation.cc:407] Connection header received
0618 22:22:25.286220 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:25.286224 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:25.286379 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:25.286488 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:25.286871 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:25.287772 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:25.288648 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:25.289058 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:25.289148 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:25.291762 (+ 2614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:22:25.291770 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:22:25.291781 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:22:25.291801 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:22:25.293809 (+ 2008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:25.294327 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:25.294336 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:25.294340 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:25.294437 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:25.294712 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:25.294717 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:25.294720 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:25.295083 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:22:25.295234 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0618 22:22:25.295488 (+ 254us) server_negotiation.cc:299] Negotiation successful
0618 22:22:25.295662 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245,"thread_start_us":152,"threads_started":1}
I20260618 22:22:25.296072 23121 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.125" port: 35773 } attempt: 3
I20260618 22:22:25.298120 23196 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.124" port: 36847 } has no permanent_uuid. Determining permanent_uuid...
I20260618 22:22:25.298931 23199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:25.298616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:36847 (local address 127.0.0.1:41518)
0618 22:22:25.298738 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0618 22:22:25.298829 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:36847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260618 22:22:25.299224 23145 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.124:36847: connect: Connection refused (error 111)
I20260618 22:22:25.300210 23199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:25.299945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:36847 (local address 127.0.0.1:41528)
0618 22:22:25.300072 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0618 22:22:25.300133 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:36847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260618 22:22:25.300964 23196 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.124:36847: Network error: Client connection negotiation failed: client connection to 127.0.98.124:36847: connect: Connection refused (error 111)
Jun 18 22:22:25 dist-test-slave-g6r4 krb5kdc[23040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821345, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for kudu/127.0.98.125@KRBTEST.COM
I20260618 22:22:25.305841 23125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:25.296533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:35773 (local address 127.0.0.1:60388)
0618 22:22:25.296663 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0618 22:22:25.296680 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0618 22:22:25.296772 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:22:25.297105 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:22:25.297109 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:22:25.297311 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:22:25.297451 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:22:25.297460 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:22:25.298645 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:22:25.298649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:22:25.299996 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:22:25.300008 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:22:25.300605 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:22:25.300610 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:22:25.300729 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:25.301255 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:22:25.301276 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:22:25.302604 (+ 1328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:22:25.304430 (+ 1826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:22:25.304433 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:22:25.304436 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:22:25.304629 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:22:25.304917 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:22:25.304922 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:22:25.304924 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:22:25.305004 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:22:25.305358 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:22:25.305362 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:22:25.305456 (+ 94us) client_negotiation.cc:769] Sending connection context
0618 22:22:25.305555 (+ 99us) client_negotiation.cc:240] Negotiation successful
0618 22:22:25.305689 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260618 22:22:25.305840 23201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:25.296603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60388 (local address 127.0.98.125:35773)
0618 22:22:25.296747 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:25.296751 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:25.296768 (+ 17us) server_negotiation.cc:407] Connection header received
0618 22:22:25.296913 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:25.296918 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:25.296973 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:25.297058 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:25.297587 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:25.298506 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:25.300117 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:25.300477 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:25.300611 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:25.302830 (+ 2219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:22:25.302833 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:22:25.302835 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:22:25.302856 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:22:25.304296 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:25.304723 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:25.304728 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:25.304730 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:25.304830 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:25.305090 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:25.305094 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:25.305096 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:25.305261 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:22:25.305357 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0618 22:22:25.305604 (+ 247us) server_negotiation.cc:299] Negotiation successful
0618 22:22:25.305705 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260618 22:22:25.306797 23121 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.124" port: 36847 } has no permanent_uuid. Determining permanent_uuid...
I20260618 22:22:25.307428 23125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:25.307224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:36847 (local address 127.0.0.1:41544)
0618 22:22:25.307330 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0618 22:22:25.307373 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:36847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260618 22:22:25.308030 23125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:25.307814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:36847 (local address 127.0.0.1:41554)
0618 22:22:25.307939 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0618 22:22:25.307981 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:36847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260618 22:22:25.308168 23121 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.124:36847: Network error: Client connection negotiation failed: client connection to 127.0.98.124:36847: connect: Connection refused (error 111)
I20260618 22:22:25.339329 23121 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.124" port: 36847 } attempt: 1
I20260618 22:22:25.340128 23125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:25.339845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:36847 (local address 127.0.0.1:41566)
0618 22:22:25.339986 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0618 22:22:25.340053 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:36847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260618 22:22:25.340924 23125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:25.340725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:36847 (local address 127.0.0.1:41570)
0618 22:22:25.340828 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0618 22:22:25.340875 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:36847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260618 22:22:25.341091 23121 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.124:36847: Network error: Client connection negotiation failed: client connection to 127.0.98.124:36847: connect: Connection refused (error 111)
I20260618 22:22:25.350270 23196 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.124" port: 36847 } attempt: 1
I20260618 22:22:25.350989 23199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:25.350748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:36847 (local address 127.0.0.1:41580)
0618 22:22:25.350875 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0618 22:22:25.350927 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:36847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260618 22:22:25.351747 23199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:25.351559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:36847 (local address 127.0.0.1:41586)
0618 22:22:25.351655 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0618 22:22:25.351698 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:36847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260618 22:22:25.351935 23196 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.124:36847: Network error: Client connection negotiation failed: client connection to 127.0.98.124:36847: connect: Connection refused (error 111)
W20260618 22:22:25.400427 23202 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260618 22:22:25.400687 23202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 22:22:25.400736 23202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 22:22:25.400815 23202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 22:22:25.404740 23202 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 22:22:25.404790 23202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 22:22:25.404809 23202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 22:22:25.404826 23202 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 22:22:25.404879 23202 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 22:22:25.409970 23202 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.84:34565
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821065380026-393-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821065380026-393-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.126:36343,127.0.98.125:35773,127.0.98.124:36847
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821065380026-393-0/krb5kdc/kudu_127.0.98.124.keytab
--principal=kudu/127.0.98.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.124:36847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821065380026-393-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23202
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821065380026-393-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 18 Jun 2026 21:46:10 UTC on ccycloud.quasar-wldxur.root.comops.site
build id 4454738
I20260618 22:22:25.410954 23202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 22:22:25.411934 23202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260618 22:22:25.416263 23121 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.124" port: 36847 } attempt: 2
I20260618 22:22:25.417021 23125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:25.416711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:36847 (local address 127.0.0.1:41594)
0618 22:22:25.416863 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0618 22:22:25.416933 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:36847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260618 22:22:25.417689 23125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:25.417492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:36847 (local address 127.0.0.1:41608)
0618 22:22:25.417588 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0618 22:22:25.417637 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:36847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260618 22:22:25.417812 23121 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.124:36847: Network error: Client connection negotiation failed: client connection to 127.0.98.124:36847: connect: Connection refused (error 111)
I20260618 22:22:25.420125 23196 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.124" port: 36847 } attempt: 2
W20260618 22:22:25.420346 23209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 22:22:25.420437 23211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:22:25.420617 23202 server_base.cc:1034] running on GCE node
W20260618 22:22:25.420372 23208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 22:22:25.420929 23199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:25.420628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:36847 (local address 127.0.0.1:41620)
0618 22:22:25.420782 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0618 22:22:25.420840 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:36847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260618 22:22:25.421514 23202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 22:22:25.421550 23199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:25.421339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:36847 (local address 127.0.0.1:41630)
0618 22:22:25.421454 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0618 22:22:25.421497 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.124:36847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260618 22:22:25.421702 23196 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.124:36847: Network error: Client connection negotiation failed: client connection to 127.0.98.124:36847: connect: Connection refused (error 111)
I20260618 22:22:25.422675 23202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 22:22:25.423851 23202 hybrid_clock.cc:648] HybridClock initialized: now 1781821345423837 us; error 36 us; skew 500 ppm
Jun 18 22:22:25 dist-test-slave-g6r4 krb5kdc[23040](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821345, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 22:22:25.426465 23202 init.cc:377] Logged in from keytab as kudu/127.0.98.124@KRBTEST.COM (short username kudu)
I20260618 22:22:25.427258 23202 webserver.cc:460] Webserver started at http://127.0.98.124:33359/ using document root <none> and password file <none>
I20260618 22:22:25.427776 23202 fs_manager.cc:359] Metadata directory not provided
I20260618 22:22:25.427858 23202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 22:22:25.428081 23202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 22:22:25.430083 23202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821065380026-393-0/minicluster-data/master-2/data/instance:
uuid: "3a88a790d1d6497ab7ffe06d126eee24"
format_stamp: "Formatted at 2026-06-18 22:22:25 on dist-test-slave-g6r4"
server_key: "0f57600bb2bfd6fbf5b5865afe184a40"
server_key_iv: "c560ca94c1a74bec412e35b256ad7195"
server_key_version: "encryptionkey@0"
I20260618 22:22:25.430639 23202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821065380026-393-0/minicluster-data/master-2/wal/instance:
uuid: "3a88a790d1d6497ab7ffe06d126eee24"
format_stamp: "Formatted at 2026-06-18 22:22:25 on dist-test-slave-g6r4"
server_key: "0f57600bb2bfd6fbf5b5865afe184a40"
server_key_iv: "c560ca94c1a74bec412e35b256ad7195"
server_key_version: "encryptionkey@0"
I20260618 22:22:25.434325 23202 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.002s
I20260618 22:22:25.436658 23218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 22:22:25.437777 23202 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260618 22:22:25.437947 23202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821065380026-393-0/minicluster-data/master-2/data,/tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821065380026-393-0/minicluster-data/master-2/wal
uuid: "3a88a790d1d6497ab7ffe06d126eee24"
format_stamp: "Formatted at 2026-06-18 22:22:25 on dist-test-slave-g6r4"
server_key: "0f57600bb2bfd6fbf5b5865afe184a40"
server_key_iv: "c560ca94c1a74bec412e35b256ad7195"
server_key_version: "encryptionkey@0"
I20260618 22:22:25.438050 23202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821065380026-393-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821065380026-393-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821065380026-393-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 22:22:25.447858 23202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 22:22:25.450505 23202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 22:22:25.450681 23202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 22:22:25.457136 23202 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.124:36847
I20260618 22:22:25.457414 23270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.124:36847 every 8 connection(s)
I20260618 22:22:25.458129 23202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821065380026-393-0/minicluster-data/master-2/data/info.pb
I20260618 22:22:25.460638 23271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 22:22:25.464077 23271 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36343 } has no permanent_uuid. Determining permanent_uuid...
I20260618 22:22:25.466061 393 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu as pid 23202
I20260618 22:22:25.466192 393 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskIMAyAx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781821065380026-393-0/minicluster-data/master-2/wal/instance
I20260618 22:22:25.466521 393 external_mini_cluster.cc:1428] Setting key 257d4a219895fcd1df9fac70d432606a
Jun 18 22:22:25 dist-test-slave-g6r4 krb5kdc[23040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821345, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.124@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
Jun 18 22:22:25 dist-test-slave-g6r4 krb5kdc[23040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.124@KRBTEST.COM
I20260618 22:22:25.477479 23124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:25.465694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48408 (local address 127.0.98.126:36343)
0618 22:22:25.465839 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:25.465849 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:25.466781 (+ 932us) server_negotiation.cc:407] Connection header received
0618 22:22:25.467921 (+ 1140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:25.467923 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:25.467963 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:25.468026 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:25.468934 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:25.469419 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:25.470532 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:25.470863 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:25.470940 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:25.473408 (+ 2468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:22:25.473412 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:22:25.473415 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:22:25.473446 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:22:25.475189 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:25.475740 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:25.475743 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:25.475745 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:25.475830 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:25.476192 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:25.476197 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:25.476199 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:25.476472 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:22:25.476561 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0618 22:22:25.477039 (+ 478us) server_negotiation.cc:299] Negotiation successful
0618 22:22:25.477176 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260618 22:22:25.478137 23274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:25.467065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41644 (local address 127.0.98.124:36847)
0618 22:22:25.467330 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:25.467343 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:25.467507 (+ 164us) server_negotiation.cc:407] Connection header received
0618 22:22:25.467725 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:25.467746 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:25.467989 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:25.468171 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:25.468872 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:25.469922 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:25.470788 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:25.471303 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:25.471426 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:25.473890 (+ 2464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:22:25.473933 (+ 43us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:22:25.473947 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:22:25.473972 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:22:25.475734 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:25.476234 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:25.476242 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:25.476244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:25.476319 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:25.476588 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:25.476591 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:25.476593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:25.476821 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:22:25.476986 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0618 22:22:25.477216 (+ 230us) server_negotiation.cc:299] Negotiation successful
0618 22:22:25.477339 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":180,"thread_start_us":87,"threads_started":1}
I20260618 22:22:25.478135 23273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:25.466152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.126:36343 (local address 127.0.0.1:48408)
0618 22:22:25.466627 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0618 22:22:25.466681 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0618 22:22:25.467725 (+ 1044us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:22:25.468092 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:22:25.468103 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:22:25.468534 (+ 431us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:22:25.468772 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:22:25.468802 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:22:25.469518 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:22:25.469522 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:22:25.470427 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:22:25.470435 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:22:25.470984 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:22:25.470987 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:22:25.471195 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:25.471807 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:22:25.471821 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:22:25.473260 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:22:25.475347 (+ 2087us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:22:25.475355 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:22:25.475361 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:22:25.475651 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:22:25.475918 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:22:25.475920 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:22:25.475922 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:22:25.476090 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:22:25.476569 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:22:25.476578 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:22:25.476804 (+ 226us) client_negotiation.cc:769] Sending connection context
0618 22:22:25.476979 (+ 175us) client_negotiation.cc:240] Negotiation successful
0618 22:22:25.477276 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"mutex_wait_us":3,"thread_start_us":126,"threads_started":1}
I20260618 22:22:25.479485 23271 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.125" port: 35773 } has no permanent_uuid. Determining permanent_uuid...
Jun 18 22:22:25 dist-test-slave-g6r4 krb5kdc[23040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821345, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.124@KRBTEST.COM for kudu/127.0.98.125@KRBTEST.COM
Jun 18 22:22:25 dist-test-slave-g6r4 krb5kdc[23040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.126@KRBTEST.COM
I20260618 22:22:25.487816 23273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:25.479966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.125:35773 (local address 127.0.0.1:60394)
0618 22:22:25.480115 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0618 22:22:25.480126 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 22:22:25.480201 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:22:25.480661 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:22:25.480664 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:22:25.480813 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:22:25.480897 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:22:25.480901 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:22:25.481694 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:22:25.481697 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:22:25.482299 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:22:25.482307 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:22:25.482778 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:22:25.482782 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:22:25.482885 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:25.483297 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:22:25.483308 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:22:25.484795 (+ 1487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:22:25.486416 (+ 1621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:22:25.486421 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:22:25.486423 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:22:25.486612 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:22:25.486875 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:22:25.486877 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:22:25.486879 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:22:25.486959 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:22:25.487335 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:22:25.487337 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:22:25.487424 (+ 87us) client_negotiation.cc:769] Sending connection context
0618 22:22:25.487532 (+ 108us) client_negotiation.cc:240] Negotiation successful
0618 22:22:25.487648 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86}
I20260618 22:22:25.487824 23201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:25.480080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60394 (local address 127.0.98.125:35773)
0618 22:22:25.480447 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:25.480452 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:25.480467 (+ 15us) server_negotiation.cc:407] Connection header received
0618 22:22:25.480513 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:25.480516 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:25.480561 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:25.480636 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:25.481007 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:25.481600 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:25.482411 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:25.482686 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:25.482760 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:25.484919 (+ 2159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:22:25.484925 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:22:25.484931 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:22:25.484962 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:22:25.486312 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:25.486703 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:25.486708 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:25.486710 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:25.486784 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:25.487048 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:25.487051 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:25.487053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:25.487218 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:22:25.487347 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0618 22:22:25.487583 (+ 236us) server_negotiation.cc:299] Negotiation successful
0618 22:22:25.487691 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281}
I20260618 22:22:25.488292 23124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:25.480320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48424 (local address 127.0.98.126:36343)
0618 22:22:25.480422 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:25.480425 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:25.480443 (+ 18us) server_negotiation.cc:407] Connection header received
0618 22:22:25.480523 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:25.480525 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:25.480556 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:25.480629 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:25.481268 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:25.481752 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:25.482551 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:25.482770 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:25.482873 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:25.485009 (+ 2136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:22:25.485012 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:22:25.485013 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:22:25.485033 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:22:25.486587 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:25.486990 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:25.486995 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:25.486997 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:25.487073 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:25.487345 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:25.487348 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:25.487350 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:25.487509 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:22:25.487723 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0618 22:22:25.487851 (+ 128us) server_negotiation.cc:299] Negotiation successful
0618 22:22:25.488139 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260618 22:22:25.488598 23271 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.124" port: 36847 } has no permanent_uuid. Determining permanent_uuid...
Jun 18 22:22:25 dist-test-slave-g6r4 krb5kdc[23040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821345, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.124@KRBTEST.COM for kudu/127.0.98.124@KRBTEST.COM
Jun 18 22:22:25 dist-test-slave-g6r4 krb5kdc[23040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.125@KRBTEST.COM
I20260618 22:22:25.496517 23273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:25.489039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:36847 (local address 127.0.0.1:41648)
0618 22:22:25.489134 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0618 22:22:25.489146 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 22:22:25.489216 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:22:25.489470 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:22:25.489472 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:22:25.489622 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:22:25.489676 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:22:25.489680 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:22:25.490417 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:22:25.490424 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:22:25.490906 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:22:25.490912 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:22:25.491351 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:22:25.491354 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:22:25.491476 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:25.491905 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:22:25.491918 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:22:25.493172 (+ 1254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:22:25.495091 (+ 1919us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:22:25.495097 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:22:25.495101 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:22:25.495323 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:22:25.495566 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:22:25.495569 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:22:25.495571 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:22:25.495661 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:22:25.496058 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:22:25.496061 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:22:25.496150 (+ 89us) client_negotiation.cc:769] Sending connection context
0618 22:22:25.496255 (+ 105us) client_negotiation.cc:240] Negotiation successful
0618 22:22:25.496357 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":22}
I20260618 22:22:25.496626 23274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:25.489176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41648 (local address 127.0.98.124:36847)
0618 22:22:25.489282 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:25.489286 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:25.489299 (+ 13us) server_negotiation.cc:407] Connection header received
0618 22:22:25.489333 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:25.489336 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:25.489385 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:25.489462 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:25.489774 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:25.490317 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:25.491011 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:25.491250 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:25.491410 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:25.493300 (+ 1890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:22:25.493304 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:22:25.493307 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:22:25.493331 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:22:25.494990 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:25.495416 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:25.495418 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:25.495419 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:25.495485 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:25.495746 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:25.495749 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:25.495751 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:25.495962 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:22:25.496065 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0618 22:22:25.496319 (+ 254us) server_negotiation.cc:299] Negotiation successful
0618 22:22:25.496433 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":30}
I20260618 22:22:25.497467 23201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:25.489364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60410 (local address 127.0.98.125:35773)
0618 22:22:25.489457 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:25.489463 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:25.489480 (+ 17us) server_negotiation.cc:407] Connection header received
0618 22:22:25.489591 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:25.489594 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:25.489630 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:25.489713 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:25.490317 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:25.490770 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:25.491468 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:25.491690 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:25.491791 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:25.494048 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:22:25.494051 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:22:25.494053 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:22:25.494072 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:22:25.495624 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:25.496058 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:25.496061 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:25.496063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:25.496138 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:25.496468 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:25.496471 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:25.496473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:25.496690 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:22:25.496903 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0618 22:22:25.497182 (+ 279us) server_negotiation.cc:299] Negotiation successful
0618 22:22:25.497298 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260618 22:22:25.500036 23271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a88a790d1d6497ab7ffe06d126eee24: Bootstrap starting.
I20260618 22:22:25.500751 393 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 22:22:25.502589 23271 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a88a790d1d6497ab7ffe06d126eee24: Neither blocks nor log segments found. Creating new log.
I20260618 22:22:25.501969 23095 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "38ea137015de482794155db583074c26"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48424
I20260618 22:22:25.503381 23271 log.cc:824] T 00000000000000000000000000000000 P 3a88a790d1d6497ab7ffe06d126eee24: Log is configured to *not* fsync() on all Append() calls
I20260618 22:22:25.505218 23271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a88a790d1d6497ab7ffe06d126eee24: No bootstrap required, opened a new log
I20260618 22:22:25.509228 23271 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a88a790d1d6497ab7ffe06d126eee24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ea137015de482794155db583074c26" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36343 } } peers { permanent_uuid: "d0e004157c4e47c8af9a41fba626f831" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 35773 } } peers { permanent_uuid: "3a88a790d1d6497ab7ffe06d126eee24" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 36847 } }
I20260618 22:22:25.509476 23271 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a88a790d1d6497ab7ffe06d126eee24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a88a790d1d6497ab7ffe06d126eee24, State: Initialized, Role: FOLLOWER
I20260618 22:22:25.509804 23271 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a88a790d1d6497ab7ffe06d126eee24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ea137015de482794155db583074c26" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36343 } } peers { permanent_uuid: "d0e004157c4e47c8af9a41fba626f831" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 35773 } } peers { permanent_uuid: "3a88a790d1d6497ab7ffe06d126eee24" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 36847 } }
I20260618 22:22:25.510305 23277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a88a790d1d6497ab7ffe06d126eee24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ea137015de482794155db583074c26" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36343 } } peers { permanent_uuid: "d0e004157c4e47c8af9a41fba626f831" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 35773 } } peers { permanent_uuid: "3a88a790d1d6497ab7ffe06d126eee24" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 36847 } } }
I20260618 22:22:25.510434 23277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a88a790d1d6497ab7ffe06d126eee24 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:22:25.511158 23271 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a88a790d1d6497ab7ffe06d126eee24 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:22:25.512790 23196 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.124" port: 36847 } attempt: 3
W20260618 22:22:25.516220 23288 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3a88a790d1d6497ab7ffe06d126eee24: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 22:22:25.516299 23288 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 22:22:25.517037 23121 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.124" port: 36847 } attempt: 3
Jun 18 22:22:25 dist-test-slave-g6r4 krb5kdc[23040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821345, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.125@KRBTEST.COM for kudu/127.0.98.124@KRBTEST.COM
I20260618 22:22:25.515033 23202 master_runner.cc:186] Error getting master registration for 127.0.98.126:36343: OK, instance_id { permanent_uuid: "38ea137015de482794155db583074c26" instance_seqno: 1781821345056546 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260618 22:22:25.518745 23202 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260618 22:22:25.522652 23199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:25.513226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:36847 (local address 127.0.0.1:41650)
0618 22:22:25.513351 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0618 22:22:25.513368 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0618 22:22:25.513459 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:22:25.513725 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:22:25.513729 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:22:25.513951 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:22:25.514055 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:22:25.514062 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:22:25.514994 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:22:25.514997 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:22:25.515835 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:22:25.515845 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:22:25.516538 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:22:25.516542 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:22:25.516637 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:25.517238 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:22:25.517252 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:22:25.518795 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:22:25.520941 (+ 2146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:22:25.520945 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:22:25.520947 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:22:25.521197 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:22:25.521525 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:22:25.521529 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:22:25.521531 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:22:25.521656 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:22:25.522153 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:22:25.522156 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:22:25.522255 (+ 99us) client_negotiation.cc:769] Sending connection context
0618 22:22:25.522350 (+ 95us) client_negotiation.cc:240] Negotiation successful
0618 22:22:25.522481 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260618 22:22:25.523146 23274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:25.513395 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41650 (local address 127.0.98.124:36847)
0618 22:22:25.513496 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:25.513502 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:25.513519 (+ 17us) server_negotiation.cc:407] Connection header received
0618 22:22:25.513556 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:25.513560 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:25.513607 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:25.513701 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:25.514181 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:25.514876 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:25.515976 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:25.516408 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:25.516532 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:25.518928 (+ 2396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:22:25.518931 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:22:25.518933 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:22:25.518952 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:22:25.520810 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:25.521320 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:25.521324 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:25.521326 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:25.521422 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:25.521760 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:25.521763 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:25.521765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:25.522029 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:22:25.522820 (+ 791us) server_negotiation.cc:1035] Waiting for connection context
0618 22:22:25.522900 (+ 80us) server_negotiation.cc:299] Negotiation successful
0618 22:22:25.523004 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
Jun 18 22:22:25 dist-test-slave-g6r4 krb5kdc[23040](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781821345, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.126@KRBTEST.COM for kudu/127.0.98.124@KRBTEST.COM
I20260618 22:22:25.526364 23196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0e004157c4e47c8af9a41fba626f831: Bootstrap starting.
I20260618 22:22:25.528823 23196 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0e004157c4e47c8af9a41fba626f831: Neither blocks nor log segments found. Creating new log.
I20260618 22:22:25.528991 23125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:25.517363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.124:36847 (local address 127.0.0.1:41654)
0618 22:22:25.517509 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0618 22:22:25.517525 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 22:22:25.517592 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 22:22:25.519292 (+ 1700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 22:22:25.519297 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 22:22:25.519575 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0618 22:22:25.519664 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:22:25.519671 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:22:25.521113 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:22:25.521123 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:22:25.521762 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 22:22:25.521772 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 22:22:25.522415 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 22:22:25.522418 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 22:22:25.522515 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:25.523196 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 22:22:25.523212 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0618 22:22:25.524740 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 22:22:25.527370 (+ 2630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:22:25.527374 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:22:25.527376 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:22:25.527616 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:22:25.527952 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 22:22:25.527955 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 22:22:25.527957 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 22:22:25.528054 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 22:22:25.528482 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 22:22:25.528485 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 22:22:25.528582 (+ 97us) client_negotiation.cc:769] Sending connection context
0618 22:22:25.528689 (+ 107us) client_negotiation.cc:240] Negotiation successful
0618 22:22:25.528817 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79}
I20260618 22:22:25.529490 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:25.517856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41654 (local address 127.0.98.124:36847)
0618 22:22:25.519074 (+ 1218us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:25.519083 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:25.519096 (+ 13us) server_negotiation.cc:407] Connection header received
0618 22:22:25.519129 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:25.519131 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:25.519171 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:25.519259 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:25.520182 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:25.520996 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:25.521874 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:25.522277 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:25.522401 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:25.524882 (+ 2481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:22:25.524885 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:22:25.524887 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:22:25.524910 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:22:25.527246 (+ 2336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:25.527733 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:25.527736 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:25.527738 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:25.527849 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:25.528160 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:25.528162 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:25.528164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:25.528374 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:22:25.528469 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0618 22:22:25.528773 (+ 304us) server_negotiation.cc:299] Negotiation successful
0618 22:22:25.528953 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1153,"thread_start_us":119,"threads_started":1}
I20260618 22:22:25.529728 23196 log.cc:824] T 00000000000000000000000000000000 P d0e004157c4e47c8af9a41fba626f831: Log is configured to *not* fsync() on all Append() calls
I20260618 22:22:25.531983 23196 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0e004157c4e47c8af9a41fba626f831: No bootstrap required, opened a new log
I20260618 22:22:25.532279 23121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38ea137015de482794155db583074c26: Bootstrap starting.
I20260618 22:22:25.534680 23121 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38ea137015de482794155db583074c26: Neither blocks nor log segments found. Creating new log.
I20260618 22:22:25.535494 23121 log.cc:824] T 00000000000000000000000000000000 P 38ea137015de482794155db583074c26: Log is configured to *not* fsync() on all Append() calls
I20260618 22:22:25.536270 23196 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0e004157c4e47c8af9a41fba626f831 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ea137015de482794155db583074c26" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36343 } } peers { permanent_uuid: "d0e004157c4e47c8af9a41fba626f831" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 35773 } } peers { permanent_uuid: "3a88a790d1d6497ab7ffe06d126eee24" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 36847 } }
I20260618 22:22:25.536514 23196 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0e004157c4e47c8af9a41fba626f831 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0e004157c4e47c8af9a41fba626f831, State: Initialized, Role: FOLLOWER
I20260618 22:22:25.536827 23196 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0e004157c4e47c8af9a41fba626f831 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ea137015de482794155db583074c26" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36343 } } peers { permanent_uuid: "d0e004157c4e47c8af9a41fba626f831" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 35773 } } peers { permanent_uuid: "3a88a790d1d6497ab7ffe06d126eee24" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 36847 } }
I20260618 22:22:25.537353 23293 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0e004157c4e47c8af9a41fba626f831 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ea137015de482794155db583074c26" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36343 } } peers { permanent_uuid: "d0e004157c4e47c8af9a41fba626f831" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 35773 } } peers { permanent_uuid: "3a88a790d1d6497ab7ffe06d126eee24" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 36847 } } }
I20260618 22:22:25.537572 23121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38ea137015de482794155db583074c26: No bootstrap required, opened a new log
I20260618 22:22:25.537652 23293 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0e004157c4e47c8af9a41fba626f831 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:22:25.538132 23196 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0e004157c4e47c8af9a41fba626f831 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:22:25.540669 23121 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38ea137015de482794155db583074c26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ea137015de482794155db583074c26" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36343 } } peers { permanent_uuid: "d0e004157c4e47c8af9a41fba626f831" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 35773 } } peers { permanent_uuid: "3a88a790d1d6497ab7ffe06d126eee24" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 36847 } }
I20260618 22:22:25.540993 23121 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38ea137015de482794155db583074c26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38ea137015de482794155db583074c26, State: Initialized, Role: FOLLOWER
I20260618 22:22:25.541316 23121 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38ea137015de482794155db583074c26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ea137015de482794155db583074c26" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36343 } } peers { permanent_uuid: "d0e004157c4e47c8af9a41fba626f831" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 35773 } } peers { permanent_uuid: "3a88a790d1d6497ab7ffe06d126eee24" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 36847 } }
I20260618 22:22:25.541934 23305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38ea137015de482794155db583074c26 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ea137015de482794155db583074c26" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36343 } } peers { permanent_uuid: "d0e004157c4e47c8af9a41fba626f831" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 35773 } } peers { permanent_uuid: "3a88a790d1d6497ab7ffe06d126eee24" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 36847 } } }
I20260618 22:22:25.542098 23305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38ea137015de482794155db583074c26 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:22:25.542614 23121 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38ea137015de482794155db583074c26 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 22:22:25.542659 23095 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38ea137015de482794155db583074c26 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260618 22:22:25.542759 23095 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38ea137015de482794155db583074c26 [term 0 FOLLOWER]: Advancing to term 1
W20260618 22:22:25.543713 23304 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d0e004157c4e47c8af9a41fba626f831: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 22:22:25.543841 23304 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 22:22:25.544044 23095 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38ea137015de482794155db583074c26 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ea137015de482794155db583074c26" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36343 } } peers { permanent_uuid: "d0e004157c4e47c8af9a41fba626f831" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 35773 } } peers { permanent_uuid: "3a88a790d1d6497ab7ffe06d126eee24" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 36847 } }
I20260618 22:22:25.544966 23095 leader_election.cc:288] T 00000000000000000000000000000000 P 38ea137015de482794155db583074c26 [CANDIDATE]: Term 1 election: Requested vote from peers d0e004157c4e47c8af9a41fba626f831 (127.0.98.125:35773), 3a88a790d1d6497ab7ffe06d126eee24 (127.0.98.124:36847)
I20260618 22:22:25.541576 23128 master_runner.cc:186] Error getting master registration for 127.0.98.126:36343: OK, instance_id { permanent_uuid: "38ea137015de482794155db583074c26" instance_seqno: 1781821345056546 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260618 22:22:25.545383 23128 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260618 22:22:25.545651 23170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "38ea137015de482794155db583074c26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d0e004157c4e47c8af9a41fba626f831"
I20260618 22:22:25.545857 23170 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0e004157c4e47c8af9a41fba626f831 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:22:25.546667 23245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "38ea137015de482794155db583074c26" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "3a88a790d1d6497ab7ffe06d126eee24"
I20260618 22:22:25.546907 23245 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a88a790d1d6497ab7ffe06d126eee24 [term 0 FOLLOWER]: Advancing to term 1
I20260618 22:22:25.547163 23170 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d0e004157c4e47c8af9a41fba626f831 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38ea137015de482794155db583074c26 in term 1.
I20260618 22:22:25.547648 23069 leader_election.cc:302] T 00000000000000000000000000000000 P 38ea137015de482794155db583074c26 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38ea137015de482794155db583074c26, d0e004157c4e47c8af9a41fba626f831; no voters:
I20260618 22:22:25.548213 23245 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 3a88a790d1d6497ab7ffe06d126eee24 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38ea137015de482794155db583074c26 in term 1.
W20260618 22:22:25.552248 23317 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 38ea137015de482794155db583074c26: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 22:22:25.552423 23317 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 22:22:25.552623 23305 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38ea137015de482794155db583074c26 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 22:22:25.552632 23053 master_runner.cc:263] No leader master found from master 38ea137015de482794155db583074c26
I20260618 22:22:25.552706 23053 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260618 22:22:25.552834 23305 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38ea137015de482794155db583074c26 [term 1 LEADER]: Becoming Leader. State: Replica: 38ea137015de482794155db583074c26, State: Running, Role: LEADER
W20260618 22:22:25.552928 23074 tablet.cc:2369] T 00000000000000000000000000000000 P 38ea137015de482794155db583074c26: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 22:22:25.553114 23305 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38ea137015de482794155db583074c26 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ea137015de482794155db583074c26" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36343 } } peers { permanent_uuid: "d0e004157c4e47c8af9a41fba626f831" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 35773 } } peers { permanent_uuid: "3a88a790d1d6497ab7ffe06d126eee24" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 36847 } }
I20260618 22:22:25.555531 23323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38ea137015de482794155db583074c26 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38ea137015de482794155db583074c26. Latest consensus state: current_term: 1 leader_uuid: "38ea137015de482794155db583074c26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ea137015de482794155db583074c26" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36343 } } peers { permanent_uuid: "d0e004157c4e47c8af9a41fba626f831" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 35773 } } peers { permanent_uuid: "3a88a790d1d6497ab7ffe06d126eee24" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 36847 } } }
I20260618 22:22:25.555658 23323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38ea137015de482794155db583074c26 [sys.catalog]: This master's current role is: LEADER
I20260618 22:22:25.555964 23328 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 22:22:25.556636 23328 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 22:22:25.567966 23124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:25.554326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48438 (local address 127.0.98.126:36343)
0618 22:22:25.554433 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:25.554437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:25.558188 (+ 3751us) server_negotiation.cc:407] Connection header received
0618 22:22:25.558350 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:25.558353 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:25.558400 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:25.558470 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:25.559443 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:25.560256 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:25.561351 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:25.561765 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:25.561864 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:25.563949 (+ 2085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:22:25.563955 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:22:25.563980 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:22:25.564006 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:22:25.566054 (+ 2048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:25.566578 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:25.566584 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:25.566589 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:25.566673 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:25.566994 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:25.566997 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:25.566999 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:25.567236 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:22:25.567356 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0618 22:22:25.567693 (+ 337us) server_negotiation.cc:299] Negotiation successful
0618 22:22:25.567817 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260618 22:22:25.570607 23290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:25.554721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41664 (local address 127.0.98.124:36847)
0618 22:22:25.554832 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:25.554835 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:25.558537 (+ 3702us) server_negotiation.cc:407] Connection header received
0618 22:22:25.558703 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:25.558706 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:25.558751 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:25.558801 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:25.559945 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:25.560669 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:25.561777 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:25.562140 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:25.562234 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:25.564591 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:22:25.564598 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:22:25.564603 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:22:25.564634 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:22:25.566544 (+ 1910us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:25.568389 (+ 1845us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:25.568392 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:25.568394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:25.568479 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:25.569631 (+ 1152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:25.569637 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:25.569642 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:25.569857 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:22:25.570019 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0618 22:22:25.570321 (+ 302us) server_negotiation.cc:299] Negotiation successful
0618 22:22:25.570438 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260618 22:22:25.577178 23201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:25.555208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60414 (local address 127.0.98.125:35773)
0618 22:22:25.555308 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:25.555313 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:25.559332 (+ 4019us) server_negotiation.cc:407] Connection header received
0618 22:22:25.565028 (+ 5696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:25.565035 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:25.565088 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:25.565146 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:25.569678 (+ 4532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:25.570425 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:25.572100 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:25.572337 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:25.572447 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:25.573738 (+ 1291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 22:22:25.573742 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 22:22:25.573744 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 22:22:25.573774 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0618 22:22:25.575380 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:25.575858 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:25.575861 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:25.575863 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:25.575943 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 22:22:25.576230 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 22:22:25.576234 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 22:22:25.576236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0618 22:22:25.576442 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 22:22:25.576537 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0618 22:22:25.576896 (+ 359us) server_negotiation.cc:299] Negotiation successful
0618 22:22:25.577013 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260618 22:22:25.577834 23245 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 3a88a790d1d6497ab7ffe06d126eee24 [term 1 FOLLOWER]: Refusing update from remote peer 38ea137015de482794155db583074c26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:22:25.578841 23323 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 38ea137015de482794155db583074c26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3a88a790d1d6497ab7ffe06d126eee24" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 36847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:22:25.582609 23277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a88a790d1d6497ab7ffe06d126eee24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38ea137015de482794155db583074c26. Latest consensus state: current_term: 1 leader_uuid: "38ea137015de482794155db583074c26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ea137015de482794155db583074c26" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36343 } } peers { permanent_uuid: "d0e004157c4e47c8af9a41fba626f831" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 35773 } } peers { permanent_uuid: "3a88a790d1d6497ab7ffe06d126eee24" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 36847 } } }
I20260618 22:22:25.582723 23277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a88a790d1d6497ab7ffe06d126eee24 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:22:25.583223 23170 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d0e004157c4e47c8af9a41fba626f831 [term 1 FOLLOWER]: Refusing update from remote peer 38ea137015de482794155db583074c26: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 22:22:25.584813 23323 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 38ea137015de482794155db583074c26 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0e004157c4e47c8af9a41fba626f831" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 35773 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 22:22:25.588387 23293 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0e004157c4e47c8af9a41fba626f831 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38ea137015de482794155db583074c26. Latest consensus state: current_term: 1 leader_uuid: "38ea137015de482794155db583074c26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ea137015de482794155db583074c26" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36343 } } peers { permanent_uuid: "d0e004157c4e47c8af9a41fba626f831" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 35773 } } peers { permanent_uuid: "3a88a790d1d6497ab7ffe06d126eee24" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 36847 } } }
I20260618 22:22:25.588512 23293 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0e004157c4e47c8af9a41fba626f831 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:22:25.589844 23323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38ea137015de482794155db583074c26 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "38ea137015de482794155db583074c26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ea137015de482794155db583074c26" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36343 } } peers { permanent_uuid: "d0e004157c4e47c8af9a41fba626f831" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 35773 } } peers { permanent_uuid: "3a88a790d1d6497ab7ffe06d126eee24" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 36847 } } }
I20260618 22:22:25.589998 23323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38ea137015de482794155db583074c26 [sys.catalog]: This master's current role is: LEADER
I20260618 22:22:25.590221 23330 mvcc.cc:205] Tried to move back new op lower bound from 7298340231478538240 to 7298340231388659712. Current Snapshot: MvccSnapshot[applied={T|T < 7298340231478538240}]
I20260618 22:22:25.590546 23277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a88a790d1d6497ab7ffe06d126eee24 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "38ea137015de482794155db583074c26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ea137015de482794155db583074c26" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36343 } } peers { permanent_uuid: "d0e004157c4e47c8af9a41fba626f831" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 35773 } } peers { permanent_uuid: "3a88a790d1d6497ab7ffe06d126eee24" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 36847 } } }
I20260618 22:22:25.590646 23277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a88a790d1d6497ab7ffe06d126eee24 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:22:25.591353 23329 mvcc.cc:205] Tried to move back new op lower bound from 7298340231478538240 to 7298340231388659712. Current Snapshot: MvccSnapshot[applied={T|T < 7298340231478538240}]
I20260618 22:22:25.591693 23305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38ea137015de482794155db583074c26 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "38ea137015de482794155db583074c26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ea137015de482794155db583074c26" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36343 } } peers { permanent_uuid: "d0e004157c4e47c8af9a41fba626f831" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 35773 } } peers { permanent_uuid: "3a88a790d1d6497ab7ffe06d126eee24" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 36847 } } }
I20260618 22:22:25.591804 23305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38ea137015de482794155db583074c26 [sys.catalog]: This master's current role is: LEADER
I20260618 22:22:25.592507 23293 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0e004157c4e47c8af9a41fba626f831 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "38ea137015de482794155db583074c26" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38ea137015de482794155db583074c26" member_type: VOTER last_known_addr { host: "127.0.98.126" port: 36343 } } peers { permanent_uuid: "d0e004157c4e47c8af9a41fba626f831" member_type: VOTER last_known_addr { host: "127.0.98.125" port: 35773 } } peers { permanent_uuid: "3a88a790d1d6497ab7ffe06d126eee24" member_type: VOTER last_known_addr { host: "127.0.98.124" port: 36847 } } }
I20260618 22:22:25.592630 23293 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0e004157c4e47c8af9a41fba626f831 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 22:22:25.595512 23328 catalog_manager.cc:1347] Generated new cluster ID: 69de4379378843978e9eb7158f2692ed
I20260618 22:22:25.595571 23328 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 22:22:25.604595 23332 mvcc.cc:205] Tried to move back new op lower bound from 7298340231478538240 to 7298340231388659712. Current Snapshot: MvccSnapshot[applied={T|T < 7298340231478538240}]
I20260618 22:22:25.608600 23328 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 22:22:25.609809 23328 catalog_manager.cc:1504] Loading token signing keys...
I20260618 22:22:25.625382 23328 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 38ea137015de482794155db583074c26: Generated new TSK 0
I20260618 22:22:25.626199 23328 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260618 22:22:25.659936 23124 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:22:25.657042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48454 (local address 127.0.98.126:36343)
0618 22:22:25.657173 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:25.657178 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:25.658926 (+ 1748us) server_negotiation.cc:407] Connection header received
0618 22:22:25.659113 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:25.659117 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:25.659178 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:25.659230 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:25.659810 (+ 580us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48454: BlockingRecv error: recv got EOF from 127.0.0.1:48454 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260618 22:22:25.661329 23290 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:22:25.657721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41680 (local address 127.0.98.124:36847)
0618 22:22:25.657827 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:25.657831 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:25.660527 (+ 2696us) server_negotiation.cc:407] Connection header received
0618 22:22:25.660686 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:25.660689 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:25.660732 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:25.660780 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:25.661217 (+ 437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41680: BlockingRecv error: recv got EOF from 127.0.0.1:41680 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260618 22:22:25.662901 23201 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:22:25.658376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60422 (local address 127.0.98.125:35773)
0618 22:22:25.658482 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:25.658486 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:25.661943 (+ 3457us) server_negotiation.cc:407] Connection header received
0618 22:22:25.662192 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:25.662196 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:25.662244 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:25.662293 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:25.662795 (+ 502us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60422: BlockingRecv error: recv got EOF from 127.0.0.1:60422 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260618 22:22:25.667968 23201 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:22:25.665650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60430 (local address 127.0.98.125:35773)
0618 22:22:25.665750 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:25.665753 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:25.667081 (+ 1328us) server_negotiation.cc:407] Connection header received
0618 22:22:25.667294 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:25.667297 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:25.667365 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:25.667413 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:25.667879 (+ 466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60430: BlockingRecv error: recv got EOF from 127.0.0.1:60430 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260618 22:22:25.672680 23201 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:22:25.671413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60432 (local address 127.0.98.125:35773)
0618 22:22:25.671503 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:25.671507 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:25.671909 (+ 402us) server_negotiation.cc:407] Connection header received
0618 22:22:25.672069 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:25.672072 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:25.672114 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:25.672172 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:25.672594 (+ 422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60432: BlockingRecv error: recv got EOF from 127.0.0.1:60432 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260618 22:22:25.678012 23201 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:22:25.676715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60436 (local address 127.0.98.125:35773)
0618 22:22:25.676800 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:25.676804 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:25.677215 (+ 411us) server_negotiation.cc:407] Connection header received
0618 22:22:25.677381 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:25.677384 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:25.677429 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:25.677476 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:25.677931 (+ 455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60436: BlockingRecv error: recv got EOF from 127.0.0.1:60436 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260618 22:22:25.685331 23201 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:22:25.684017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60448 (local address 127.0.98.125:35773)
0618 22:22:25.684142 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:25.684146 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:25.684553 (+ 407us) server_negotiation.cc:407] Connection header received
0618 22:22:25.684717 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:25.684720 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:25.684762 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:25.684808 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:25.685245 (+ 437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60448: BlockingRecv error: recv got EOF from 127.0.0.1:60448 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260618 22:22:25.697031 23201 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:22:25.695739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60458 (local address 127.0.98.125:35773)
0618 22:22:25.695831 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:25.695836 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:25.696258 (+ 422us) server_negotiation.cc:407] Connection header received
0618 22:22:25.696420 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:25.696423 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:25.696465 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:25.696510 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:25.696939 (+ 429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60458: BlockingRecv error: recv got EOF from 127.0.0.1:60458 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260618 22:22:25.717253 23201 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:22:25.715620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60472 (local address 127.0.98.125:35773)
0618 22:22:25.715737 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:25.715742 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:25.716266 (+ 524us) server_negotiation.cc:407] Connection header received
0618 22:22:25.716483 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:25.716487 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:25.716540 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:25.716592 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:25.717125 (+ 533us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60472: BlockingRecv error: recv got EOF from 127.0.0.1:60472 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260618 22:22:25.753723 23201 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:22:25.752106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60486 (local address 127.0.98.125:35773)
0618 22:22:25.752238 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:25.752242 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:25.752781 (+ 539us) server_negotiation.cc:407] Connection header received
0618 22:22:25.752973 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:25.752977 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:25.753028 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:25.753077 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:25.753603 (+ 526us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60486: BlockingRecv error: recv got EOF from 127.0.0.1:60486 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260618 22:22:25.821318 23201 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:22:25.820306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60502 (local address 127.0.98.125:35773)
0618 22:22:25.820469 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:25.820472 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:25.820523 (+ 51us) server_negotiation.cc:407] Connection header received
0618 22:22:25.820651 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:25.820653 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:25.820701 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:25.820775 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:25.821182 (+ 407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60502: BlockingRecv error: recv got EOF from 127.0.0.1:60502 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260618 22:22:25.952708 23201 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:22:25.951381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60510 (local address 127.0.98.125:35773)
0618 22:22:25.951553 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:25.951558 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:25.951714 (+ 156us) server_negotiation.cc:407] Connection header received
0618 22:22:25.951894 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:25.951898 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:25.951952 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:25.952042 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:25.952564 (+ 522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60510: BlockingRecv error: recv got EOF from 127.0.0.1:60510 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260618 22:22:26.211939 23201 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 22:22:26.210782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60520 (local address 127.0.98.125:35773)
0618 22:22:26.210912 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:26.210918 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:26.211016 (+ 98us) server_negotiation.cc:407] Connection header received
0618 22:22:26.211180 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:26.211184 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:26.211242 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:26.211334 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0618 22:22:26.211823 (+ 489us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60520: BlockingRecv error: recv got EOF from 127.0.0.1:60520 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260618 22:22:26.517501 23288 catalog_manager.cc:1259] Loaded cluster ID: 69de4379378843978e9eb7158f2692ed
I20260618 22:22:26.517598 23288 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 3a88a790d1d6497ab7ffe06d126eee24: loading cluster ID for follower catalog manager: success
I20260618 22:22:26.519237 23288 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 3a88a790d1d6497ab7ffe06d126eee24: acquiring CA information for follower catalog manager: success
I20260618 22:22:26.520370 23288 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 3a88a790d1d6497ab7ffe06d126eee24: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260618 22:22:26.544991 23304 catalog_manager.cc:1259] Loaded cluster ID: 69de4379378843978e9eb7158f2692ed
I20260618 22:22:26.545060 23304 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d0e004157c4e47c8af9a41fba626f831: loading cluster ID for follower catalog manager: success
I20260618 22:22:26.546705 23304 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d0e004157c4e47c8af9a41fba626f831: acquiring CA information for follower catalog manager: success
I20260618 22:22:26.547785 23304 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d0e004157c4e47c8af9a41fba626f831: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260618 22:22:26.730024 23402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:26.726009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60530 (local address 127.0.98.125:35773)
0618 22:22:26.726288 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:26.726294 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:26.726397 (+ 103us) server_negotiation.cc:407] Connection header received
0618 22:22:26.726574 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:26.726577 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:26.726717 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:26.726834 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:22:26.727085 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:26.727629 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:26.728496 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:26.728735 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:26.728847 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:26.729167 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:22:26.729584 (+ 417us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:22:26.729727 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0618 22:22:26.729770 (+ 43us) server_negotiation.cc:299] Negotiation successful
0618 22:22:26.729824 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":80,"threads_started":1}
I20260618 22:22:26.736297 23408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:26.733020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41692 (local address 127.0.98.124:36847)
0618 22:22:26.733255 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:26.733259 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:26.733272 (+ 13us) server_negotiation.cc:407] Connection header received
0618 22:22:26.733370 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:26.733372 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:26.733477 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:26.733571 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:22:26.733757 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:26.734306 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:26.735066 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:26.735288 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:26.735384 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:26.735594 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:22:26.736005 (+ 411us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:22:26.736094 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0618 22:22:26.736145 (+ 51us) server_negotiation.cc:299] Negotiation successful
0618 22:22:26.736188 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":75,"threads_started":1}
I20260618 22:22:26.743532 23402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:26.739144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60534 (local address 127.0.98.125:35773)
0618 22:22:26.739277 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:26.739281 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:26.739536 (+ 255us) server_negotiation.cc:407] Connection header received
0618 22:22:26.739716 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:26.739719 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:26.739763 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:26.739847 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:22:26.740064 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:26.740760 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:26.741805 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:26.742176 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:26.742290 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:26.743021 (+ 731us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:22:26.743095 (+ 74us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:22:26.743158 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0618 22:22:26.743305 (+ 147us) server_negotiation.cc:299] Negotiation successful
0618 22:22:26.743390 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260618 22:22:26.743646 23408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:26.739693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41700 (local address 127.0.98.124:36847)
0618 22:22:26.740005 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:26.740010 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:26.740023 (+ 13us) server_negotiation.cc:407] Connection header received
0618 22:22:26.740059 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:26.740061 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:26.740101 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:26.740175 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:22:26.740408 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:26.741142 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:26.742502 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:26.742811 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:26.742936 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:26.743177 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:22:26.743239 (+ 62us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:22:26.743320 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0618 22:22:26.743458 (+ 138us) server_negotiation.cc:299] Negotiation successful
0618 22:22:26.743503 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":242}
I20260618 22:22:26.744071 23415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 22:22:26.739179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48460 (local address 127.0.98.126:36343)
0618 22:22:26.739479 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0618 22:22:26.739486 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 22:22:26.739502 (+ 16us) server_negotiation.cc:407] Connection header received
0618 22:22:26.739677 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 22:22:26.739682 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 22:22:26.739839 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 22:22:26.739944 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 22:22:26.740350 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:26.741187 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:26.742305 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 22:22:26.742613 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 22:22:26.742701 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 22:22:26.742983 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 22:22:26.743299 (+ 316us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 22:22:26.743463 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0618 22:22:26.743746 (+ 283us) server_negotiation.cc:299] Negotiation successful
0618 22:22:26.743798 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":84,"threads_started":1}
I20260618 22:22:26.746841 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 23053
I20260618 22:22:26.753212 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 23128
I20260618 22:22:26.758975 393 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskIMAyAx/build/debug/bin/kudu with pid 23202
2026-06-18T22:22:26Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3035 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6162 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (281370 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST